# Fast Sentence Transformers
This repository contains code to run faster `sentence-transformers` using tools like quantization and `ONNX`. Just run your model much faster, while a lot of memory. There is not much to it!
[](https://github.com/Pandora-Intelligence/fast-sentence-transformers/actions/workflows/python-package.yml)
[](https://github.com/pandora-intelligence/fast-sentence-transformers/releases)
[](https://pypi.org/project/fast-sentence-transformers/)
[](https://pypi.org/project/fast-sentence-transformers/)
[](https://github.com/ambv/black)
# Install
```bash
pip install fast-sentence-transformers
```
Or for GPU support.
```bash
pip install fast-sentence-transformers[gpu]
```
# Quickstart
```python
from fast_sentence_transformers import FastSentenceTransformer as SentenceTransformer
# use any sentence-transformer
encoder = SentenceTransformer("all-MiniLM-L6-v2", device="cpu", quantize=True)
encoder.encode("Hello hello, hey, hello hello")
encoder.encode(["Life is too short to eat bad food!"] * 2)
```
# Benchmark
Indicative benchmark for CPU usage with smallest and largest model on `sentence-transformers`. Note, ONNX doesn't have GPU support for quantization yet.
| model | Type | default | ONNX | ONNX+quantized | ONNX+GPU |
| ------------------------------------- | ------ | ------- | ---- | -------------- | -------- |
| paraphrase-albert-small-v2 | memory | 1x | 1x | 1x | 1x |
| | speed | 1x | 2x | 5x | 20x |
| paraphrase-multilingual-mpnet-base-v2 | memory | 1x | 1x | 4x | 4x |
| | speed | 1x | 2x | 5x | 20x |
# Shout-Out
This package heavily leans on `sentence-transformers` and `txtai`.
Raw data
{
"_id": null,
"home_page": "https://github.com/pandora-intelligence/fast-sentence-transformers",
"name": "fast-sentence-transformers",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<3.12",
"maintainer_email": "",
"keywords": "sentence-transformerx,ONNX,NLP",
"author": "David Berenstein",
"author_email": "david.m.berenstein@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/7b/c3/d8911a83ea89808bacb066f778db78ef60cd02363ed2e31fd93ffc5f7b1b/fast_sentence_transformers-0.4.1.tar.gz",
"platform": null,
"description": "# Fast Sentence Transformers\nThis repository contains code to run faster `sentence-transformers` using tools like quantization and `ONNX`. Just run your model much faster, while a lot of memory. There is not much to it!\n\n[](https://github.com/Pandora-Intelligence/fast-sentence-transformers/actions/workflows/python-package.yml)\n[](https://github.com/pandora-intelligence/fast-sentence-transformers/releases)\n[](https://pypi.org/project/fast-sentence-transformers/)\n[](https://pypi.org/project/fast-sentence-transformers/)\n[](https://github.com/ambv/black)\n\n# Install\n```bash\npip install fast-sentence-transformers\n```\nOr for GPU support.\n```bash\npip install fast-sentence-transformers[gpu]\n```\n\n# Quickstart\n\n```python\n\nfrom fast_sentence_transformers import FastSentenceTransformer as SentenceTransformer\n\n# use any sentence-transformer\nencoder = SentenceTransformer(\"all-MiniLM-L6-v2\", device=\"cpu\", quantize=True)\n\nencoder.encode(\"Hello hello, hey, hello hello\")\nencoder.encode([\"Life is too short to eat bad food!\"] * 2)\n```\n\n# Benchmark\nIndicative benchmark for CPU usage with smallest and largest model on `sentence-transformers`. Note, ONNX doesn't have GPU support for quantization yet.\n\n| model | Type | default | ONNX | ONNX+quantized | ONNX+GPU |\n| ------------------------------------- | ------ | ------- | ---- | -------------- | -------- |\n| paraphrase-albert-small-v2 | memory | 1x | 1x | 1x | 1x |\n| | speed | 1x | 2x | 5x | 20x |\n| paraphrase-multilingual-mpnet-base-v2 | memory | 1x | 1x | 4x | 4x |\n| | speed | 1x | 2x | 5x | 20x |\n\n# Shout-Out\n\nThis package heavily leans on `sentence-transformers` and `txtai`.\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "This repository contains code to run faster sentence-transformers using tools like quantization, ONNX and pruning.",
"version": "0.4.1",
"split_keywords": [
"sentence-transformerx",
"onnx",
"nlp"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "836919293e3d201ce4f46ea56559fac1f76d81484e5d7fc75739a7585e2e2098",
"md5": "0c7535f2aba353588d05024a69ad8a2c",
"sha256": "83d47d623c2ad5b5d2bf7d5fffcfeb32a994df82be71414fd60c0e849026a94b"
},
"downloads": -1,
"filename": "fast_sentence_transformers-0.4.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0c7535f2aba353588d05024a69ad8a2c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<3.12",
"size": 22235,
"upload_time": "2023-01-11T22:12:48",
"upload_time_iso_8601": "2023-01-11T22:12:48.177018Z",
"url": "https://files.pythonhosted.org/packages/83/69/19293e3d201ce4f46ea56559fac1f76d81484e5d7fc75739a7585e2e2098/fast_sentence_transformers-0.4.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7bc3d8911a83ea89808bacb066f778db78ef60cd02363ed2e31fd93ffc5f7b1b",
"md5": "ed2babe98827b0796c98476ec9a9defe",
"sha256": "d4d963b5495cb070701e67517f89c7d9a0f11179281e1d9c4f4dcafd230ada0f"
},
"downloads": -1,
"filename": "fast_sentence_transformers-0.4.1.tar.gz",
"has_sig": false,
"md5_digest": "ed2babe98827b0796c98476ec9a9defe",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<3.12",
"size": 16865,
"upload_time": "2023-01-11T22:12:49",
"upload_time_iso_8601": "2023-01-11T22:12:49.807426Z",
"url": "https://files.pythonhosted.org/packages/7b/c3/d8911a83ea89808bacb066f778db78ef60cd02363ed2e31fd93ffc5f7b1b/fast_sentence_transformers-0.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-01-11 22:12:49",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "pandora-intelligence",
"github_project": "fast-sentence-transformers",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "fast-sentence-transformers"
}
JSON
