# espnet_onnx
[](https://pypi.org/project/espnet-onnx/)
**ESPnet without PyTorch!**
Utility library to easily export, quantize, and optimize espnet models to onnx format.
There is no need to install PyTorch or ESPnet on your machine if you already have exported files!
## espnet_onnx demo in Colab
Now demonstration notebook is available in google colab!
- Simple ASR demo: [](https://colab.research.google.com/github/espnet/espnet_onnx/blob/master/demo/simple_asr_demo.ipynb)
- Simple TTS demo: [](https://colab.research.google.com/github/espnet/espnet_onnx/blob/master/demo/tts_onnx_demo.ipynb)
## Install
1. `espnet_onnx` can be installed with pip
```shell
pip install espnet_onnx
```
2. If you want to export pretrained model, you need to install `torch>=1.11.0`, `espnet`, `espnet_model_zoo`, `onnx` additionally.
`onnx==1.12.0` might cause some errors. If you got an error while inference or exporting, please consider downgrading the onnx version.
#### Install guide for developers
1. Clone this repository.
```shell
git clone git@github.com:espnet/espnet_onnx.git
```
2. Create virtual environment.
```shell
cd tools
make venv export
```
3. Activate virtual environment and install torch if required.
```shell
. tools/venv/bin/activate
# Please reference official installation guide of PyTorch.
pip install torch
```
4. Clone the s3prl repository and install with pip.
```shell
cd tools
git clone https://github.com/s3prl/s3prl
cd s3prl
pip install .
```
5. Install warp_transducer for developing transducer model.
```shell
cd tools
git clone --single-branch --branch espnet_v1.1 https://github.com/b-flo/warp-transducer.git
cd warp-transducer
mkdir build
# Please set WITH_OMP to ON or OFF.
cd build && cmake -DWITH_OMP="ON" .. && make
cd pytorch_binding && python3 -m pip install -e .
```
6. If you want to develop optimization, you also need to develop onnxruntime. Please clone the onnxruntime repository.
7. Since espnet==202308(latest on v0.2.0 release) requires `protobuf<=3.20.1` while the latest onnx requires `protobuf>=3.20.2`, you may get error with installation.
In this case, first, install the espnet==202308, update protobuf to 3.20.3, and then install other libraries.
## Usage
#### Export models
1. `espnet_onnx` can export pretrained model published on `espnet_model_zoo`. By default, exported files will be stored in `${HOME}/.cache/espnet_onnx/<tag_name>`.
```python
from espnet2.bin.asr_inference import Speech2Text
from espnet_onnx.export import ASRModelExport
m = ASRModelExport()
# download with espnet_model_zoo and export from pretrained model
m.export_from_pretrained('<tag name>', quantize=True)
# export from trained model
speech2text = Speech2Text(args)
m.export(speech2text, '<tag name>', quantize=True)
```
2. You can export pretrained model from zipped file. The zipped file should contain `meta.yaml`.
```python
from espnet_onnx.export import ASRModelExport
m = ASRModelExport()
m.export_from_zip(
'path/to/the/zipfile',
tag_name='tag_name_for_zipped_model',
quantize=True
)
```
3. You can set some configuration for export. The available configurations are shown in the details for each models.
- [Details for ASR models and configurations](./docs/ASRModelDetail.md)
- [Details for TTS models and configurations](./docs/TTSModelDetail.md)
- [Details for optimization configurations](./doc/optimize)
```python
from espnet_onnx.export import ASRModelExport
m = ASRModelExport()
# Set maximum sequence length to 3000
m.set_export_config(max_seq_len=3000)
m.export_from_zip(
'path/to/the/zipfile',
tag_name='tag_name_for_zipped_model',
)
```
1. You can easily optimize your model by using the `optimize` option. If you want to fully optimize your model, you need to install the custom version of onnxruntime from [here](https://github.com/espnet/espnet_onnx/releases/download/custom_ort_v1.11.1-espnet_onnx/onnxruntime-1.11.1_espnet_onnx-cp38-cp38-linux_x86_64.whl). Please read [this document](./docs/Optimization.md) for more detail.
```python
from espnet_onnx.export import ASRModelExport
m = ASRModelExport()
m.export_from_zip(
'path/to/the/zipfile',
tag_name='tag_name_for_zipped_model',
optimize=True,
quantize=True
)
```
5. You can export model from the command line.
```shell
python -m espnet_onnx.export \
--model_type asr \
--input ${path_to_zip} \
--tag transformer_lm \
--apply_optimize \
--apply_quantize
```
#### Inference
1. For inference, `tag_name` or `model_dir` is used to load onnx file. `tag_name` has to be defined in `tag_config.yaml`
```python
import librosa
from espnet_onnx import Speech2Text
speech2text = Speech2Text(tag_name='<tag name>')
# speech2text = Speech2Text(model_dir='path to the onnx directory')
y, sr = librosa.load('sample.wav', sr=16000)
nbest = speech2text(y)
```
3. For streaming asr, you can use `StreamingSpeech2Text` class. The speech length should be the same as `StreamingSpeech2Text.hop_size`
```python
from espnet_onnx import StreamingSpeech2Text
stream_asr = StreamingSpeech2Text(tag_name)
# start streaming asr
stream_asr.start()
while streaming:
wav = <some code to get wav>
assert len(wav) == stream_asr.hop_size
stream_text = stream_asr(wav)[0][0]
# You can get non-streaming asr result with end function
nbest = stream_asr.end()
```
You can also simulate streaming model with your wav file with `simulate` function. Passing `True` as the second argument will show the streaming text as the following code.
```python
import librosa
from espnet_onnx import StreamingSpeech2Text
stream_asr = StreamingSpeech2Text(tag_name)
y, sr = librosa.load('path/to/wav', sr=16000)
nbest = stream_asr.simulate(y, True)
# Processing audio with 6 processes.
# Result at position 0 :
# Result at position 1 :
# Result at position 2 : this
# Result at position 3 : this is
# Result at position 4 : this is a
# Result at position 5 : this is a
print(nbest[0][0])
# 'this is a pen'
```
4. If you installed the custom version of onnxruntime, you can run optimized model for inference. You don't have to change any code from the above. If the model was optimized, then espnet_onnx would automatically load the optimized version.
5. You can use only hubert model for your frontend.
```python
from espnet_onnx.export import ASRModelExport
# export your model
tag_name = 'ESPnet pretrained model with hubert'
m = ASRModelExport()
m.export_from_pretrained(tag_name, optimize=True)
# load only the frontend model
from espnet_onnx.asr.frontend import Frontend
frontend = Frontend.get_frontend(tag_name)
# use the model in your application
import librosa
y, sr = librosa.load('wav file')
# y: (B, T)
# y_len: (B,)
feats = frontend(y[None,:], np.array([len(y)]))
```
6. If you installed `torch` in your environment, you can use frontend in your training.
```python
from espnet_onnx.asr.frontend import TorchFrontend
frontend = TorchFrontend.get_frontend(tag_name) # load pretrained frontend model
# use the model while training
import librosa
y, sr = librosa.load('wav file')
# You need to place your data on GPU,
# and specify the output shape in tuple
y = torch.Tensor(y).unsqueeze(0).to('cuda') # (1, wav_length)
output_shape = (batch_size, feat_length, feats_dims)
feats = frontend(y, y.size(1), output_shape)
```
#### Text2Speech inference
1. You can export TTS models as ASR models.
```python
from espnet2.bin.tts_inference import Text2Speech
from espnet_onnx.export import TTSModelExport
m = TTSModelExport()
# download with espnet_model_zoo and export from pretrained model
m.export_from_pretrained('<tag name>', quantize=True)
# export from trained model
text2speech = Text2Speech(args)
m.export(text2speech, '<tag name>', quantize=True)
```
2. You can generate wav files with just simply using the Text2Speech class.
```python
from espnet_onnx import Text2Speech
tag_name = 'kan-bayashi/ljspeech_vits'
text2speech = Text2Speech(tag_name, use_quantized=True)
text = 'Hello world!'
output_dict = text2speech(text) # inference with onnx model.
wav = output_dict['wav']
```
## How to use GPU on espnet_onnx
**Install dependency.**
First, we need `onnxruntime-gpu` library, instead of `onnxruntime`. Please follow [this article](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html) to select and install the correct version of `onnxruntime-gpu`, depending on your CUDA version.
**Inference on GPU**
Now you can speedup the inference speed with GPU. All you need is to select the correct providers, and give it to the `Speech2Text` or `StreamingSpeech2Text` instance. See [this article](https://onnxruntime.ai/docs/execution-providers/) for more information about providers.
```python
import librosa
from espnet_onnx import Speech2Text
PROVIDERS = ['CUDAExecutionProvider']
tag_name = 'some_tag_name'
speech2text = Speech2Text(
tag_name,
providers=PROVIDERS
)
y, sr = librosa.load('path/to/wav', sr=16000)
nbest = speech2text(y) # runs on GPU.
```
Note that some quantized models are not supported for GPU computation. If you got an error with quantized model, please try not-quantized model.
## Changes from ESPNet
To avoid the cache problem, I modified some scripts from the original espnet implementation.
1. Add `<blank>` before `<sos>`
2. Give some `torch.zeros()` arrays to the model.
3. Remove the first token in post process. (remove `blank`)
4. Replace `make_pad_mask` into new implementation, which can be converted into onnx format.
5. Removed `extend_pe()` from positional encoding module. The length of `pe` is 512 by default.
## Supported Archs
ASR: [Supported architecture for ASR](./docs/ASRSupported.md)
TTS: [Supported architecture for TTS](./docs/TTSSupported.md)
## Developer's Guide
ASR: [Developer's Guide](./docs/DeveloperGuide.md)
## References
- [ESPNet: end-to-end speech processing toolkit](https://github.com/espnet/espnet)
- [ESPNet Model Zoo](https://github.com/espnet/espnet_model_zoo)
## COPYRIGHT
Copyright (c) 2022 Maso Someki
Released under [MIT licence](https://opensource.org/licenses/mit-license.php)
## Author
Masao Someki
contact: `masao.someki@gmail.com`
Raw data
{
"_id": null,
"home_page": "https://github.com/Masao-Someki/espnet_onnx",
"name": "espnet-onnx",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7.0",
"maintainer_email": null,
"keywords": "espnet onnxruntime",
"author": "Masao Someki",
"author_email": "masao.someki@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/b1/21/5ce746419dabf2bdeac8bd6dc75d74915b35ea7e45d7b645f1dae1f96d72/espnet_onnx-0.2.1.tar.gz",
"platform": null,
"description": "# espnet_onnx\n\n\n\n[](https://pypi.org/project/espnet-onnx/)\n\n**ESPnet without PyTorch!**\n\nUtility library to easily export, quantize, and optimize espnet models to onnx format.\nThere is no need to install PyTorch or ESPnet on your machine if you already have exported files!\n\n\n## espnet_onnx demo in Colab\nNow demonstration notebook is available in google colab!\n- Simple ASR demo: [](https://colab.research.google.com/github/espnet/espnet_onnx/blob/master/demo/simple_asr_demo.ipynb)\n- Simple TTS demo: [](https://colab.research.google.com/github/espnet/espnet_onnx/blob/master/demo/tts_onnx_demo.ipynb)\n\n## Install\n\n1. `espnet_onnx` can be installed with pip\n\n```shell\npip install espnet_onnx\n```\n\n2. If you want to export pretrained model, you need to install `torch>=1.11.0`, `espnet`, `espnet_model_zoo`, `onnx` additionally.\n`onnx==1.12.0` might cause some errors. If you got an error while inference or exporting, please consider downgrading the onnx version.\n\n\n#### Install guide for developers\n\n1. Clone this repository.\n\n```shell\ngit clone git@github.com:espnet/espnet_onnx.git\n```\n\n2. Create virtual environment.\n\n```shell\ncd tools\nmake venv export\n```\n\n3. Activate virtual environment and install torch if required.\n\n```shell\n. tools/venv/bin/activate\n\n# Please reference official installation guide of PyTorch.\npip install torch\n```\n\n4. Clone the s3prl repository and install with pip.\n\n```shell\ncd tools\ngit clone https://github.com/s3prl/s3prl\ncd s3prl\npip install .\n```\n\n5. Install warp_transducer for developing transducer model.\n\n```shell\ncd tools\ngit clone --single-branch --branch espnet_v1.1 https://github.com/b-flo/warp-transducer.git\ncd warp-transducer\nmkdir build\n# Please set WITH_OMP to ON or OFF.\ncd build && cmake -DWITH_OMP=\"ON\" .. && make\ncd pytorch_binding && python3 -m pip install -e .\n```\n\n6. If you want to develop optimization, you also need to develop onnxruntime. Please clone the onnxruntime repository.\n\n7. Since espnet==202308(latest on v0.2.0 release) requires `protobuf<=3.20.1` while the latest onnx requires `protobuf>=3.20.2`, you may get error with installation.\n In this case, first, install the espnet==202308, update protobuf to 3.20.3, and then install other libraries.\n\n\n## Usage\n\n#### Export models\n\n1. `espnet_onnx` can export pretrained model published on `espnet_model_zoo`. By default, exported files will be stored in `${HOME}/.cache/espnet_onnx/<tag_name>`.\n\n```python\nfrom espnet2.bin.asr_inference import Speech2Text\nfrom espnet_onnx.export import ASRModelExport\n\nm = ASRModelExport()\n\n# download with espnet_model_zoo and export from pretrained model\nm.export_from_pretrained('<tag name>', quantize=True)\n\n# export from trained model\nspeech2text = Speech2Text(args)\nm.export(speech2text, '<tag name>', quantize=True)\n```\n\n2. You can export pretrained model from zipped file. The zipped file should contain `meta.yaml`.\n\n```python\nfrom espnet_onnx.export import ASRModelExport\n\nm = ASRModelExport()\nm.export_from_zip(\n 'path/to/the/zipfile',\n tag_name='tag_name_for_zipped_model',\n quantize=True\n)\n```\n\n3. You can set some configuration for export. The available configurations are shown in the details for each models.\n\n- [Details for ASR models and configurations](./docs/ASRModelDetail.md)\n- [Details for TTS models and configurations](./docs/TTSModelDetail.md)\n- [Details for optimization configurations](./doc/optimize)\n\n```python\nfrom espnet_onnx.export import ASRModelExport\n\nm = ASRModelExport()\n# Set maximum sequence length to 3000\nm.set_export_config(max_seq_len=3000)\nm.export_from_zip(\n 'path/to/the/zipfile',\n tag_name='tag_name_for_zipped_model',\n)\n```\n\n1. You can easily optimize your model by using the `optimize` option. If you want to fully optimize your model, you need to install the custom version of onnxruntime from [here](https://github.com/espnet/espnet_onnx/releases/download/custom_ort_v1.11.1-espnet_onnx/onnxruntime-1.11.1_espnet_onnx-cp38-cp38-linux_x86_64.whl). Please read [this document](./docs/Optimization.md) for more detail.\n\n```python\nfrom espnet_onnx.export import ASRModelExport\n\nm = ASRModelExport()\nm.export_from_zip(\n 'path/to/the/zipfile',\n tag_name='tag_name_for_zipped_model',\n optimize=True,\n quantize=True\n)\n```\n\n5. You can export model from the command line.\n\n```shell\npython -m espnet_onnx.export \\\n --model_type asr \\\n --input ${path_to_zip} \\\n --tag transformer_lm \\\n --apply_optimize \\\n --apply_quantize\n```\n\n\n\n#### Inference\n\n1. For inference, `tag_name` or `model_dir` is used to load onnx file. `tag_name` has to be defined in `tag_config.yaml`\n\n```python\nimport librosa\nfrom espnet_onnx import Speech2Text\n\nspeech2text = Speech2Text(tag_name='<tag name>')\n# speech2text = Speech2Text(model_dir='path to the onnx directory')\n\ny, sr = librosa.load('sample.wav', sr=16000)\nnbest = speech2text(y)\n```\n\n3. For streaming asr, you can use `StreamingSpeech2Text` class. The speech length should be the same as `StreamingSpeech2Text.hop_size`\n\n```python\nfrom espnet_onnx import StreamingSpeech2Text\n\nstream_asr = StreamingSpeech2Text(tag_name)\n\n# start streaming asr\nstream_asr.start()\nwhile streaming:\n wav = <some code to get wav>\n assert len(wav) == stream_asr.hop_size\n stream_text = stream_asr(wav)[0][0]\n\n# You can get non-streaming asr result with end function\nnbest = stream_asr.end()\n```\n\nYou can also simulate streaming model with your wav file with `simulate` function. Passing `True` as the second argument will show the streaming text as the following code.\n\n```python\nimport librosa\nfrom espnet_onnx import StreamingSpeech2Text\n\nstream_asr = StreamingSpeech2Text(tag_name)\ny, sr = librosa.load('path/to/wav', sr=16000)\nnbest = stream_asr.simulate(y, True)\n# Processing audio with 6 processes.\n# Result at position 0 :\n# Result at position 1 :\n# Result at position 2 : this\n# Result at position 3 : this is\n# Result at position 4 : this is a\n# Result at position 5 : this is a\nprint(nbest[0][0])\n# 'this is a pen'\n```\n\n4. If you installed the custom version of onnxruntime, you can run optimized model for inference. You don't have to change any code from the above. If the model was optimized, then espnet_onnx would automatically load the optimized version.\n\n5. You can use only hubert model for your frontend.\n\n```python\nfrom espnet_onnx.export import ASRModelExport\n\n# export your model\ntag_name = 'ESPnet pretrained model with hubert'\nm = ASRModelExport()\nm.export_from_pretrained(tag_name, optimize=True)\n\n# load only the frontend model\nfrom espnet_onnx.asr.frontend import Frontend\nfrontend = Frontend.get_frontend(tag_name)\n\n# use the model in your application\nimport librosa\ny, sr = librosa.load('wav file')\n# y: (B, T)\n# y_len: (B,)\nfeats = frontend(y[None,:], np.array([len(y)]))\n```\n\n6. If you installed `torch` in your environment, you can use frontend in your training.\n\n```python\nfrom espnet_onnx.asr.frontend import TorchFrontend\nfrontend = TorchFrontend.get_frontend(tag_name) # load pretrained frontend model\n\n# use the model while training\nimport librosa\ny, sr = librosa.load('wav file')\n\n# You need to place your data on GPU,\n# and specify the output shape in tuple\ny = torch.Tensor(y).unsqueeze(0).to('cuda') # (1, wav_length)\noutput_shape = (batch_size, feat_length, feats_dims)\nfeats = frontend(y, y.size(1), output_shape)\n```\n\n#### Text2Speech inference\n\n1. You can export TTS models as ASR models.\n\n```python\nfrom espnet2.bin.tts_inference import Text2Speech\nfrom espnet_onnx.export import TTSModelExport\n\nm = TTSModelExport()\n\n# download with espnet_model_zoo and export from pretrained model\nm.export_from_pretrained('<tag name>', quantize=True)\n\n# export from trained model\ntext2speech = Text2Speech(args)\nm.export(text2speech, '<tag name>', quantize=True)\n```\n\n2. You can generate wav files with just simply using the Text2Speech class.\n\n```python\nfrom espnet_onnx import Text2Speech\n\ntag_name = 'kan-bayashi/ljspeech_vits'\ntext2speech = Text2Speech(tag_name, use_quantized=True)\n\ntext = 'Hello world!'\noutput_dict = text2speech(text) # inference with onnx model.\nwav = output_dict['wav']\n```\n\n## How to use GPU on espnet_onnx\n\n**Install dependency.**\n\nFirst, we need `onnxruntime-gpu` library, instead of `onnxruntime`. Please follow [this article](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html) to select and install the correct version of `onnxruntime-gpu`, depending on your CUDA version.\n\n**Inference on GPU**\n\nNow you can speedup the inference speed with GPU. All you need is to select the correct providers, and give it to the `Speech2Text` or `StreamingSpeech2Text` instance. See [this article](https://onnxruntime.ai/docs/execution-providers/) for more information about providers.\n\n```python\nimport librosa\nfrom espnet_onnx import Speech2Text\n\nPROVIDERS = ['CUDAExecutionProvider']\ntag_name = 'some_tag_name'\n\nspeech2text = Speech2Text(\n tag_name,\n providers=PROVIDERS\n)\ny, sr = librosa.load('path/to/wav', sr=16000)\nnbest = speech2text(y) # runs on GPU.\n```\n\nNote that some quantized models are not supported for GPU computation. If you got an error with quantized model, please try not-quantized model.\n\n## Changes from ESPNet\n\nTo avoid the cache problem, I modified some scripts from the original espnet implementation.\n\n1. Add `<blank>` before `<sos>`\n2. Give some `torch.zeros()` arrays to the model.\n3. Remove the first token in post process. (remove `blank`)\n4. Replace `make_pad_mask` into new implementation, which can be converted into onnx format.\n\n5. Removed `extend_pe()` from positional encoding module. The length of `pe` is 512 by default.\n\n## Supported Archs\n\nASR: [Supported architecture for ASR](./docs/ASRSupported.md)\n\nTTS: [Supported architecture for TTS](./docs/TTSSupported.md)\n\n## Developer's Guide\n\nASR: [Developer's Guide](./docs/DeveloperGuide.md)\n\n## References\n\n- [ESPNet: end-to-end speech processing toolkit](https://github.com/espnet/espnet)\n- [ESPNet Model Zoo](https://github.com/espnet/espnet_model_zoo)\n\n## COPYRIGHT\n\nCopyright (c) 2022 Maso Someki\n\nReleased under [MIT licence](https://opensource.org/licenses/mit-license.php)\n\n## Author\n\nMasao Someki\n\ncontact: `masao.someki@gmail.com`\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "ONNX Wrapper for ESPnet",
"version": "0.2.1",
"project_urls": {
"Homepage": "https://github.com/Masao-Someki/espnet_onnx"
},
"split_keywords": [
"espnet",
"onnxruntime"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5f93ed3983c38ae7d16e37a73a1cd66c8ab384c682d3314c76b6acf640850406",
"md5": "a00b6ca9f09baa2ec14a2a081169bf75",
"sha256": "325a68236b2a95d11d779edd790bf0372cdcc7a52ece5e9e2a1005953c6e3338"
},
"downloads": -1,
"filename": "espnet_onnx-0.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a00b6ca9f09baa2ec14a2a081169bf75",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7.0",
"size": 144405,
"upload_time": "2024-06-16T16:06:17",
"upload_time_iso_8601": "2024-06-16T16:06:17.989595Z",
"url": "https://files.pythonhosted.org/packages/5f/93/ed3983c38ae7d16e37a73a1cd66c8ab384c682d3314c76b6acf640850406/espnet_onnx-0.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b1215ce746419dabf2bdeac8bd6dc75d74915b35ea7e45d7b645f1dae1f96d72",
"md5": "06c8d15c40f16e244bb1d9d005e51fbb",
"sha256": "32e4b375f119f10ce4b3bf088bdc9e6ba7cbf3a99b5bd577c3ddf22dba0210c5"
},
"downloads": -1,
"filename": "espnet_onnx-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "06c8d15c40f16e244bb1d9d005e51fbb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7.0",
"size": 97842,
"upload_time": "2024-06-16T16:06:20",
"upload_time_iso_8601": "2024-06-16T16:06:20.171456Z",
"url": "https://files.pythonhosted.org/packages/b1/21/5ce746419dabf2bdeac8bd6dc75d74915b35ea7e45d7b645f1dae1f96d72/espnet_onnx-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-16 16:06:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Masao-Someki",
"github_project": "espnet_onnx",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"circle": true,
"lcname": "espnet-onnx"
}
JSON
