..
Copyright 2021 Xilinx, Inc.
Copyright 2022 Advanced Micro Devices, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
AMD Inference Server
====================
The AMD Inference Server is an open-source tool to deploy your machine learning models and make them accessible to clients for inference.
Out-of-the-box, the server can support selected models that run on AMD CPUs, GPUs or FPGAs by leveraging existing libraries.
For all these models and hardware accelerators, the server presents a common user interface based on community standards so clients can make requests to any using the same API.
The server provides HTTP/REST and gRPC interfaces for clients to submit requests.
For both, there are C++ and Python bindings to simplify writing client programs.
You can also use the server backend directly using the native C++ API to write local applications.
Features
--------
* Supports client requests using **HTTP/REST, gRPC and websocket protocols** using an API based on `KServe's v2 specification <https://github.com/kserve/kserve/blob/master/docs/predict-api/v2/required_api.md>`__
* Custom applications can directly call the backend bypassing the other protocols using the **native C++ API**
* **C++ library with Python bindings** to simplify making requests to the server
* Incoming requests are transparently **batched** based on the user specifications
* Users can define how many models, and how many instances of each, to **run in parallel**
The AMD Inference Server is integrated with the following libraries out of the gate:
* TensorFlow and PyTorch models with `ZenDNN <https://developer.amd.com/zendnn/>`__ on CPUs (optimized for AMD CPUs)
* ONNX models with `MIGraphX <https://github.com/ROCmSoftwarePlatform/AMDMIGraphX>`__ on AMD GPUs
* XModel models with `Vitis AI <https://www.xilinx.com/products/design-tools/vitis/vitis-ai.html>`__ on AMD FPGAs
* A graph of computation including as pre- and post-processing can be written using `AKS <https://github.com/Xilinx/Vitis-AI/tree/bbd45838d4a93f894cfc9f232140dc65af2398d1/src/AKS>`__ on AMD FPGAs for end-to-end inference
Quick Start Deployment and Inference
------------------------------------
The following example demonstrates how to deploy the server locally and run a sample inference.
This example runs on the CPU and does not require any special hardware.
You can see a more detailed version of this example in the `quickstart <https://xilinx.github.io/inference-server/main/quickstart.html>`__.
.. code-block:: bash
# Step 1: Download the example files and create a model repository
wget https://github.com/Xilinx/inference-server/raw/main/examples/resnet50/quickstart-setup.sh
chmod +x ./quickstart-setup.sh
./quickstart-setup.sh
# Step 2: Launch the AMD Inference Server
docker run -d --net=host -v ${PWD}/model_repository:/mnt/models:rw amdih/serve:uif1.1_zendnn_amdinfer_0.3.0 amdinfer-server --enable-repository-watcher
# Step 3: Install the Python client library
pip install amdinfer
# Step 4: Send an inference request
python3 tfzendnn.py --endpoint resnet50 --image ./dog-3619020_640.jpg --labels ./imagenet_classes.txt
# Inference should print the following:
#
# Running the TF+ZenDNN example for ResNet50 in Python
# Waiting until the server is ready...
# Making inferences...
# Top 5 classes for ../../tests/assets/dog-3619020_640.jpg:
# n02112018 Pomeranian
# n02112350 keeshond
# n02086079 Pekinese, Pekingese, Peke
# n02112137 chow, chow chow
# n02113023 Pembroke, Pembroke Welsh corgi
Learn more
----------
The documentation for the AMD Inference Server is available `online <https://xilinx.github.io/inference-server/>`__.
Check out the `quickstart <https://xilinx.github.io/inference-server/main/quickstart.html>`__ online to help you get started.
Support
-------
Raise issues if you find a bug or need help.
Refer to `Contributing <https://xilinx.github.io/inference-server/main/contributing.html>`__ for more information.
License
-------
The AMD Inference Server is licensed under the terms of Apache 2.0 (see `LICENSE <https://github.com/Xilinx/inference-server/blob/main/LICENSE>`__).
The LICENSE file contains additional license information for third-party files distributed with this work.
More license information can be seen in the `dependencies <https://xilinx.github.io/inference-server/main/dependencies.html>`__.
IMPORTANT NOTICE CONCERNING OPEN-SOURCE SOFTWARE
Materials in this release may be licensed by Xilinx or third parties and may
be subject to the GNU General Public License, the GNU Lesser General License,
or other licenses.
Licenses and source files may be downloaded from:
* `libamdinfer <https://github.com/Xilinx/inference-server>`__
* `libbrotlicommon <https://rhel.pkgs.org/7/epel-x86_64/libbrotli-1.0.9-10.el7.x86_64.rpm.html>`__
* `libbrotlidec <https://rhel.pkgs.org/7/epel-x86_64/libbrotli-1.0.9-10.el7.x86_64.rpm.html>`__
* `libbrotlienc <https://rhel.pkgs.org/7/epel-x86_64/libbrotli-1.0.9-10.el7.x86_64.rpm.html>`__
* `libcares <https://github.com/c-ares/c-ares/archive/refs/tags/cares-1_14_0.tar.gz>`__
* `libcom_err <https://centos.pkgs.org/7/centos-x86_64/libcom_err-1.42.9-19.el7.x86_64.rpm.html>`__
* `libcrypto <https://centos.pkgs.org/7/centos-x86_64/openssl-devel-1.0.2k-19.el7.x86_64.rpm.html>`__
* `libdrogon <https://github.com/drogonframework/drogon/tree/v1.8.1>`__
* `libefsw <https://github.com/SpartanJ/efsw/tree/6b51944994b5c77dbd7edce66846e378a3bf4d8e>`__
* `libgssapi_krb5 <https://centos.pkgs.org/7/centos-x86_64/krb5-libs-1.15.1-50.el7.x86_64.rpm.html>`__
* `libjsoncpp <https://github.com/open-source-parsers/jsoncpp/tree/1.7.4>`__
* `libk5crypto <https://centos.pkgs.org/7/centos-x86_64/krb5-libs-1.15.1-50.el7.x86_64.rpm.html>`__
* `libkeyutils <https://centos.pkgs.org/7/centos-x86_64/keyutils-1.5.8-3.el7.x86_64.rpm.html>`__
* `libkrb5 <https://centos.pkgs.org/7/centos-x86_64/krb5-devel-1.15.1-50.el7.x86_64.rpm.html>`__
* `libkrb5support <https://centos.pkgs.org/7/centos-x86_64/krb5-devel-1.15.1-50.el7.x86_64.rpm.html>`__
* `libopencv_core <https://github.com/opencv/opencv/tree/3.4.3>`__
* `libopencv_imgcodecs <https://github.com/opencv/opencv/tree/3.4.3>`__
* `libopencv_imgproc <https://github.com/opencv/opencv/tree/3.4.3>`__
* `libossp-uuid <https://centos.pkgs.org/7/centos-x86_64/uuid-devel-1.6.2-26.el7.x86_64.rpm.html>`__
* `libpcre <https://centos.pkgs.org/7/centos-x86_64/pcre-8.32-17.el7.x86_64.rpm.html>`__
* `libprometheus-cpp-core <https://github.com/jupp0r/prometheus-cpp/archive/refs/tags/v0.12.2.tar.gz>`__
* `libprotobuf <https://github.com/protocolbuffers/protobuf/releases/download/v3.19.4/protobuf-cpp-3.19.4.tar.gz>`__
* `libselinux <https://centos.pkgs.org/7/centos-x86_64/libselinux-2.5-15.el7.x86_64.rpm.html>`__
* `libssl <https://centos.pkgs.org/7/centos-x86_64/openssl-devel-1.0.2k-19.el7.x86_64.rpm.html>`__
* `libtrantor <https://github.com/drogonframework/drogon/tree/v1.8.1>`__
Note: You are solely responsible for checking the header files and other
accompanying source files (i) provided within, in support of, or that otherwise
accompanies these materials or (ii) created from the use of third party
software and tools (and associated libraries and utilities) that are supplied
with these materials, because such header and/or source files may contain
or describe various copyright notices and license terms and conditions
governing such files, which vary from case to case based on your usage
and are beyond the control of Xilinx. You are solely responsible for
complying with the terms and conditions imposed by third parties as applicable
to your software applications created from the use of third
party software and tools (and associated libraries and utilities) that are
supplied with the materials.
Raw data
{
"_id": null,
"home_page": "https://github.com/Xilinx/inference-server",
"name": "amdinfer",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "",
"author": "Varun Sharma",
"author_email": "varun.sharma@amd.com",
"download_url": "",
"platform": null,
"description": "..\n Copyright 2021 Xilinx, Inc.\n Copyright 2022 Advanced Micro Devices, Inc.\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n\nAMD Inference Server\n====================\n\nThe AMD Inference Server is an open-source tool to deploy your machine learning models and make them accessible to clients for inference.\nOut-of-the-box, the server can support selected models that run on AMD CPUs, GPUs or FPGAs by leveraging existing libraries.\nFor all these models and hardware accelerators, the server presents a common user interface based on community standards so clients can make requests to any using the same API.\nThe server provides HTTP/REST and gRPC interfaces for clients to submit requests.\nFor both, there are C++ and Python bindings to simplify writing client programs.\nYou can also use the server backend directly using the native C++ API to write local applications.\n\nFeatures\n--------\n\n* Supports client requests using **HTTP/REST, gRPC and websocket protocols** using an API based on `KServe's v2 specification <https://github.com/kserve/kserve/blob/master/docs/predict-api/v2/required_api.md>`__\n* Custom applications can directly call the backend bypassing the other protocols using the **native C++ API**\n* **C++ library with Python bindings** to simplify making requests to the server\n* Incoming requests are transparently **batched** based on the user specifications\n* Users can define how many models, and how many instances of each, to **run in parallel**\n\nThe AMD Inference Server is integrated with the following libraries out of the gate:\n\n* TensorFlow and PyTorch models with `ZenDNN <https://developer.amd.com/zendnn/>`__ on CPUs (optimized for AMD CPUs)\n* ONNX models with `MIGraphX <https://github.com/ROCmSoftwarePlatform/AMDMIGraphX>`__ on AMD GPUs\n* XModel models with `Vitis AI <https://www.xilinx.com/products/design-tools/vitis/vitis-ai.html>`__ on AMD FPGAs\n* A graph of computation including as pre- and post-processing can be written using `AKS <https://github.com/Xilinx/Vitis-AI/tree/bbd45838d4a93f894cfc9f232140dc65af2398d1/src/AKS>`__ on AMD FPGAs for end-to-end inference\n\nQuick Start Deployment and Inference\n------------------------------------\n\nThe following example demonstrates how to deploy the server locally and run a sample inference.\nThis example runs on the CPU and does not require any special hardware.\nYou can see a more detailed version of this example in the `quickstart <https://xilinx.github.io/inference-server/main/quickstart.html>`__.\n\n.. code-block:: bash\n\n # Step 1: Download the example files and create a model repository\n wget https://github.com/Xilinx/inference-server/raw/main/examples/resnet50/quickstart-setup.sh\n chmod +x ./quickstart-setup.sh\n ./quickstart-setup.sh\n\n # Step 2: Launch the AMD Inference Server\n docker run -d --net=host -v ${PWD}/model_repository:/mnt/models:rw amdih/serve:uif1.1_zendnn_amdinfer_0.3.0 amdinfer-server --enable-repository-watcher\n\n # Step 3: Install the Python client library\n pip install amdinfer\n\n # Step 4: Send an inference request\n python3 tfzendnn.py --endpoint resnet50 --image ./dog-3619020_640.jpg --labels ./imagenet_classes.txt\n\n # Inference should print the following:\n #\n # Running the TF+ZenDNN example for ResNet50 in Python\n # Waiting until the server is ready...\n # Making inferences...\n # Top 5 classes for ../../tests/assets/dog-3619020_640.jpg:\n # n02112018 Pomeranian\n # n02112350 keeshond\n # n02086079 Pekinese, Pekingese, Peke\n # n02112137 chow, chow chow\n # n02113023 Pembroke, Pembroke Welsh corgi\n\nLearn more\n----------\n\nThe documentation for the AMD Inference Server is available `online <https://xilinx.github.io/inference-server/>`__.\n\nCheck out the `quickstart <https://xilinx.github.io/inference-server/main/quickstart.html>`__ online to help you get started.\n\nSupport\n-------\n\nRaise issues if you find a bug or need help.\nRefer to `Contributing <https://xilinx.github.io/inference-server/main/contributing.html>`__ for more information.\n\nLicense\n-------\n\nThe AMD Inference Server is licensed under the terms of Apache 2.0 (see `LICENSE <https://github.com/Xilinx/inference-server/blob/main/LICENSE>`__).\nThe LICENSE file contains additional license information for third-party files distributed with this work.\nMore license information can be seen in the `dependencies <https://xilinx.github.io/inference-server/main/dependencies.html>`__.\n\nIMPORTANT NOTICE CONCERNING OPEN-SOURCE SOFTWARE\n\nMaterials in this release may be licensed by Xilinx or third parties and may\nbe subject to the GNU General Public License, the GNU Lesser General License,\nor other licenses.\n\nLicenses and source files may be downloaded from:\n\n* `libamdinfer <https://github.com/Xilinx/inference-server>`__\n* `libbrotlicommon <https://rhel.pkgs.org/7/epel-x86_64/libbrotli-1.0.9-10.el7.x86_64.rpm.html>`__\n* `libbrotlidec <https://rhel.pkgs.org/7/epel-x86_64/libbrotli-1.0.9-10.el7.x86_64.rpm.html>`__\n* `libbrotlienc <https://rhel.pkgs.org/7/epel-x86_64/libbrotli-1.0.9-10.el7.x86_64.rpm.html>`__\n* `libcares <https://github.com/c-ares/c-ares/archive/refs/tags/cares-1_14_0.tar.gz>`__\n* `libcom_err <https://centos.pkgs.org/7/centos-x86_64/libcom_err-1.42.9-19.el7.x86_64.rpm.html>`__\n* `libcrypto <https://centos.pkgs.org/7/centos-x86_64/openssl-devel-1.0.2k-19.el7.x86_64.rpm.html>`__\n* `libdrogon <https://github.com/drogonframework/drogon/tree/v1.8.1>`__\n* `libefsw <https://github.com/SpartanJ/efsw/tree/6b51944994b5c77dbd7edce66846e378a3bf4d8e>`__\n* `libgssapi_krb5 <https://centos.pkgs.org/7/centos-x86_64/krb5-libs-1.15.1-50.el7.x86_64.rpm.html>`__\n* `libjsoncpp <https://github.com/open-source-parsers/jsoncpp/tree/1.7.4>`__\n* `libk5crypto <https://centos.pkgs.org/7/centos-x86_64/krb5-libs-1.15.1-50.el7.x86_64.rpm.html>`__\n* `libkeyutils <https://centos.pkgs.org/7/centos-x86_64/keyutils-1.5.8-3.el7.x86_64.rpm.html>`__\n* `libkrb5 <https://centos.pkgs.org/7/centos-x86_64/krb5-devel-1.15.1-50.el7.x86_64.rpm.html>`__\n* `libkrb5support <https://centos.pkgs.org/7/centos-x86_64/krb5-devel-1.15.1-50.el7.x86_64.rpm.html>`__\n* `libopencv_core <https://github.com/opencv/opencv/tree/3.4.3>`__\n* `libopencv_imgcodecs <https://github.com/opencv/opencv/tree/3.4.3>`__\n* `libopencv_imgproc <https://github.com/opencv/opencv/tree/3.4.3>`__\n* `libossp-uuid <https://centos.pkgs.org/7/centos-x86_64/uuid-devel-1.6.2-26.el7.x86_64.rpm.html>`__\n* `libpcre <https://centos.pkgs.org/7/centos-x86_64/pcre-8.32-17.el7.x86_64.rpm.html>`__\n* `libprometheus-cpp-core <https://github.com/jupp0r/prometheus-cpp/archive/refs/tags/v0.12.2.tar.gz>`__\n* `libprotobuf <https://github.com/protocolbuffers/protobuf/releases/download/v3.19.4/protobuf-cpp-3.19.4.tar.gz>`__\n* `libselinux <https://centos.pkgs.org/7/centos-x86_64/libselinux-2.5-15.el7.x86_64.rpm.html>`__\n* `libssl <https://centos.pkgs.org/7/centos-x86_64/openssl-devel-1.0.2k-19.el7.x86_64.rpm.html>`__\n* `libtrantor <https://github.com/drogonframework/drogon/tree/v1.8.1>`__\n\nNote: You are solely responsible for checking the header files and other\naccompanying source files (i) provided within, in support of, or that otherwise\naccompanies these materials or (ii) created from the use of third party\nsoftware and tools (and associated libraries and utilities) that are supplied\nwith these materials, because such header and/or source files may contain\nor describe various copyright notices and license terms and conditions\ngoverning such files, which vary from case to case based on your usage\nand are beyond the control of Xilinx. You are solely responsible for\ncomplying with the terms and conditions imposed by third parties as applicable\nto your software applications created from the use of third\nparty software and tools (and associated libraries and utilities) that are\nsupplied with the materials.\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Python client library for the AMD Inference Server: unified inference across AMD CPUs, GPUs, and FPGAs",
"version": "0.4.0",
"project_urls": {
"Bug Tracker": "https://github.com/Xilinx/inference-server/issues",
"Changelog": "https://xilinx.github.io/inference-server/main/changelog.html",
"Documentation": "https://xilinx.github.io/inference-server/main/index.html",
"Homepage": "https://github.com/Xilinx/inference-server",
"Source Code": "https://github.com/Xilinx/inference-server"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "78b25f68d8dcf510955baee14957ed74cc6e710f86494bd4b749a143d18be731",
"md5": "d69ab46592897e20e15023d09fcb941b",
"sha256": "76653c057fa09b84862942450568abd68e7a634a4791499ff17de6d75ada66a0"
},
"downloads": -1,
"filename": "amdinfer-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "d69ab46592897e20e15023d09fcb941b",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.6",
"size": 23461972,
"upload_time": "2023-09-07T17:24:03",
"upload_time_iso_8601": "2023-09-07T17:24:03.666543Z",
"url": "https://files.pythonhosted.org/packages/78/b2/5f68d8dcf510955baee14957ed74cc6e710f86494bd4b749a143d18be731/amdinfer-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c74915787aad7f4919a70f2912fcda0a60e512474b6b0353d3abcda330b1e7d5",
"md5": "eca6bde6616850fad8536006b6e1e4f2",
"sha256": "ef5723fbb81645b4181d9dbe8bccdc81f6fdabc3cf88ab757935f8c873f1fdba"
},
"downloads": -1,
"filename": "amdinfer-0.4.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "eca6bde6616850fad8536006b6e1e4f2",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.6",
"size": 23467145,
"upload_time": "2023-09-07T17:24:07",
"upload_time_iso_8601": "2023-09-07T17:24:07.325091Z",
"url": "https://files.pythonhosted.org/packages/c7/49/15787aad7f4919a70f2912fcda0a60e512474b6b0353d3abcda330b1e7d5/amdinfer-0.4.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8943a829dc916b10307ac0a326b8c3bcfbdf40085925670eea30379e2587fe87",
"md5": "8d75305f39806d153ac3c3bd78291787",
"sha256": "fbc41cb11dfcc0f2a17ef3e1cbc672cd902b618aa8adf461bc2ec1cdd0f87421"
},
"downloads": -1,
"filename": "amdinfer-0.4.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "8d75305f39806d153ac3c3bd78291787",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.6",
"size": 23462002,
"upload_time": "2023-09-07T17:24:10",
"upload_time_iso_8601": "2023-09-07T17:24:10.791026Z",
"url": "https://files.pythonhosted.org/packages/89/43/a829dc916b10307ac0a326b8c3bcfbdf40085925670eea30379e2587fe87/amdinfer-0.4.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "678804525ca37a38645fd44bbe3581ebb361371d8074852b4633c5675360600a",
"md5": "db615b2e6e3edef42408ab931d3461e2",
"sha256": "eb5ed9a2a27507a8fa319be65f16126ec8ff4d2d7744376fcbfb8c956c4ab883"
},
"downloads": -1,
"filename": "amdinfer-0.4.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "db615b2e6e3edef42408ab931d3461e2",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.6",
"size": 23462236,
"upload_time": "2023-09-07T17:24:15",
"upload_time_iso_8601": "2023-09-07T17:24:15.222126Z",
"url": "https://files.pythonhosted.org/packages/67/88/04525ca37a38645fd44bbe3581ebb361371d8074852b4633c5675360600a/amdinfer-0.4.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-07 17:24:03",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Xilinx",
"github_project": "inference-server",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "amdinfer"
}
JSON
