<a href="https://explosion.ai"><img src="https://explosion.ai/assets/img/logo.svg" width="125" height="125" align="right" /></a>
# thinc-apple-ops
Make [spaCy](https://spacy.io) and [Thinc](https://thinc.ai) **up to 8 × faster**
on macOS by calling into Apple's native libraries.
## ⏳ Install
Make sure you have [Xcode](https://developer.apple.com/xcode/) installed and
then install with `pip`:
```bash
pip install thinc-apple-ops
```
## 🏫 Motivation
Matrix multiplication is one of the primary operations in machine learning.
Since matrix multiplication is computationally expensive, using a fast matrix
multiplication implementation can speed up training and prediction
significantly.
Most linear algebra libraries provide matrix multiplication in the form of the
standardized
[BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) `gemm`
functions. The work behind scences is done by a set of matrix multiplication
kernels that are meticulously tuned for specific architectures. Matrix
multiplication kernels use architecture-specific
[SIMD](https://en.wikipedia.org/wiki/SIMD) instructions for data-level parallism
and can take factors such as cache sizes and intstruction latency into account.
[Thinc](https://github.com/explosion/thinc) uses the
[BLIS](https://github.com/flame/blis) linear algebra library, which provides
optimized matrix multiplication kernels for most x86_64 and some ARM CPUs.
Recent [Apple Silicon](https://en.wikipedia.org/wiki/Apple_silicon) CPUs, such
as the [M-series](https://en.wikipedia.org/wiki/Apple_silicon#M_series) used in
Macs, differ from traditional x86_64 and ARM CPUs in that they have a separate
matrix co-processor(s) called AMX. Since AMX is not well-documented, it is
unclear how many AMX units Apple M CPUs have. It is certain that the (single)
performance cluster of the M1 has an AMX unit and there is [empirical
evidence](https://twitter.com/danieldekok/status/1454383754512945155?s=20) that
both performance clusters of the M1 Pro/Max have an AMX unit.
Even though AMX units use a set of [undocumented
instructions](https://gist.github.com/dougallj/7a75a3be1ec69ca550e7c36dc75e0d6f),
the units can be used through Apple's
[Accelerate](https://developer.apple.com/documentation/accelerate) linear
algebra library. Since Accelerate implements the BLAS interface, it can be used
as a replacement of the BLIS library that is used by Thinc. This is where the
`thinc-apple-ops` package comes in. `thinc-apple-ops` extends the default Thinc
ops, so that `gemm` matrix multiplication from Accelerate is used in place of
the BLIS implementation of `gemm`. As a result, matrix multiplication in Thinc
is performed on the fast AMX unit(s).
## ⏱ Benchmarks
Using `thinc-apple-ops` leads to large speedups in prediction and training on
Apple Silicon Macs, as shown by the benchmarks below.
### Prediction
This first benchmark compares prediction speed of the `de_core_news_lg` spaCy
model between the M1 with and without `thinc-apple-ops`. Results for an Intel
Mac Mini and AMD Ryzen 5900X are also provided for comparison. Results are in
words per second. In this prediction benchmark, using `thinc-apple-ops` improves
performance by **4.3** times.
| *CPU* | *BLIS* | *thinc-apple-ops* | *Package power (Watt)* |
| -------------------------- | -----: | ----------------: | ---------------------: |
| Mac Mini (M1) | 6492 | 27676 | 5 |
| MacBook Air Core i5 2020 | 9790 | 10983 | 9 |
| Mac Mini Core i7 Late 2018 | 16364 | 14858 | 31 |
| AMD Ryzen 5900X | 22568 | N/A | 52 |
### Training
In the second benchmark, we compare the training speed of the `de_core_news_lg`
spaCy model (without NER). The results are in training iterations per second.
Using `thinc-apple-ops` improves training time by **3.0** times.
| *CPU* | *BLIS* | *thinc-apple-ops* | *Package power (Watt)* |
| -------------------------- | -----: | ----------------: | ---------------------: |
| Mac Mini M1 2020 | 3.34 | 10.07 | 5 |
| MacBook Air Core i5 2020 | 3.10 | 3.27 | 10 |
| Mac Mini Core i7 Late 2018 | 4.71 | 4.93 | 32 |
| AMD Ryzen 5900X | 6.53 | N/A | 53 |
Raw data
{
"_id": null,
"home_page": "https://github.com/explosion/thinc-apple-ops",
"name": "thinc-apple-ops",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "Explosion",
"author_email": "contact@explosion.ai",
"download_url": "https://files.pythonhosted.org/packages/ac/9b/bd3bc0bda5696adf82192cf6dfbebcdc99eaa83c7fda866579e489f86bde/thinc_apple_ops-0.1.4.tar.gz",
"platform": null,
"description": "<a href=\"https://explosion.ai\"><img src=\"https://explosion.ai/assets/img/logo.svg\" width=\"125\" height=\"125\" align=\"right\" /></a>\n\n# thinc-apple-ops\n\nMake [spaCy](https://spacy.io) and [Thinc](https://thinc.ai) **up to 8 × faster**\non macOS by calling into Apple's native libraries.\n\n## \u23f3 Install\n\nMake sure you have [Xcode](https://developer.apple.com/xcode/) installed and\nthen install with `pip`:\n\n```bash\npip install thinc-apple-ops\n```\n\n## \ud83c\udfeb Motivation\n\nMatrix multiplication is one of the primary operations in machine learning.\nSince matrix multiplication is computationally expensive, using a fast matrix\nmultiplication implementation can speed up training and prediction\nsignificantly.\n\nMost linear algebra libraries provide matrix multiplication in the form of the\nstandardized\n[BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) `gemm`\nfunctions. The work behind scences is done by a set of matrix multiplication\nkernels that are meticulously tuned for specific architectures. Matrix\nmultiplication kernels use architecture-specific\n[SIMD](https://en.wikipedia.org/wiki/SIMD) instructions for data-level parallism\nand can take factors such as cache sizes and intstruction latency into account.\n[Thinc](https://github.com/explosion/thinc) uses the\n[BLIS](https://github.com/flame/blis) linear algebra library, which provides\noptimized matrix multiplication kernels for most x86_64 and some ARM CPUs.\n\nRecent [Apple Silicon](https://en.wikipedia.org/wiki/Apple_silicon) CPUs, such\nas the [M-series](https://en.wikipedia.org/wiki/Apple_silicon#M_series) used in\nMacs, differ from traditional x86_64 and ARM CPUs in that they have a separate\nmatrix co-processor(s) called AMX. Since AMX is not well-documented, it is\nunclear how many AMX units Apple M CPUs have. It is certain that the (single)\nperformance cluster of the M1 has an AMX unit and there is [empirical\nevidence](https://twitter.com/danieldekok/status/1454383754512945155?s=20) that\nboth performance clusters of the M1 Pro/Max have an AMX unit.\n\n\nEven though AMX units use a set of [undocumented\ninstructions](https://gist.github.com/dougallj/7a75a3be1ec69ca550e7c36dc75e0d6f),\nthe units can be used through Apple's\n[Accelerate](https://developer.apple.com/documentation/accelerate) linear\nalgebra library. Since Accelerate implements the BLAS interface, it can be used\nas a replacement of the BLIS library that is used by Thinc. This is where the\n`thinc-apple-ops` package comes in. `thinc-apple-ops` extends the default Thinc\nops, so that `gemm` matrix multiplication from Accelerate is used in place of\nthe BLIS implementation of `gemm`. As a result, matrix multiplication in Thinc\nis performed on the fast AMX unit(s).\n\n## \u23f1 Benchmarks\n\nUsing `thinc-apple-ops` leads to large speedups in prediction and training on\nApple Silicon Macs, as shown by the benchmarks below.\n\n### Prediction\n\nThis first benchmark compares prediction speed of the `de_core_news_lg` spaCy\nmodel between the M1 with and without `thinc-apple-ops`. Results for an Intel\nMac Mini and AMD Ryzen 5900X are also provided for comparison. Results are in\nwords per second. In this prediction benchmark, using `thinc-apple-ops` improves\nperformance by **4.3** times.\n\n| *CPU* | *BLIS* | *thinc-apple-ops* | *Package power (Watt)* |\n| -------------------------- | -----: | ----------------: | ---------------------: |\n| Mac Mini (M1) | 6492 | 27676 | 5 |\n| MacBook Air Core i5 2020 | 9790 | 10983 | 9 |\n| Mac Mini Core i7 Late 2018 | 16364 | 14858 | 31 |\n| AMD Ryzen 5900X | 22568 | N/A | 52 |\n\n### Training\n\nIn the second benchmark, we compare the training speed of the `de_core_news_lg`\nspaCy model (without NER). The results are in training iterations per second.\nUsing `thinc-apple-ops` improves training time by **3.0** times.\n\n| *CPU* | *BLIS* | *thinc-apple-ops* | *Package power (Watt)* |\n| -------------------------- | -----: | ----------------: | ---------------------: |\n| Mac Mini M1 2020 | 3.34 | 10.07 | 5 |\n| MacBook Air Core i5 2020 | 3.10 | 3.27 | 10 |\n| Mac Mini Core i7 Late 2018 | 4.71 | 4.93 | 32 |\n| AMD Ryzen 5900X | 6.53 | N/A | 53 |\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Improve Thinc's performance on Apple devices with native libraries",
"version": "0.1.4",
"project_urls": {
"Homepage": "https://github.com/explosion/thinc-apple-ops"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "cdf086d6e02bb0a20c666c3c4601cd62bbb5556c14106e3c4372d657e8a467a9",
"md5": "c055b7801e7e9a3b7a7b3818e11f7c56",
"sha256": "ad03f4247f8243164d4faba79aa157899d2a3ab22e2964f01be93a9a889895ee"
},
"downloads": -1,
"filename": "thinc_apple_ops-0.1.4-cp310-cp310-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "c055b7801e7e9a3b7a7b3818e11f7c56",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.7",
"size": 161702,
"upload_time": "2023-09-22T14:50:02",
"upload_time_iso_8601": "2023-09-22T14:50:02.399820Z",
"url": "https://files.pythonhosted.org/packages/cd/f0/86d6e02bb0a20c666c3c4601cd62bbb5556c14106e3c4372d657e8a467a9/thinc_apple_ops-0.1.4-cp310-cp310-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "35dc38fc5d39b3e16e43b7f2eb976e3f64c9b9413fd7dd46660a2606327afcf4",
"md5": "e2bb40470fc661689221207e65958969",
"sha256": "ec3bd84e0c5ef5d9ee1a10de07c5bfd477f936029ebc698b3b418f82963dbd0b"
},
"downloads": -1,
"filename": "thinc_apple_ops-0.1.4-cp310-cp310-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "e2bb40470fc661689221207e65958969",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.7",
"size": 154569,
"upload_time": "2023-09-22T14:50:04",
"upload_time_iso_8601": "2023-09-22T14:50:04.789452Z",
"url": "https://files.pythonhosted.org/packages/35/dc/38fc5d39b3e16e43b7f2eb976e3f64c9b9413fd7dd46660a2606327afcf4/thinc_apple_ops-0.1.4-cp310-cp310-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "97dff0e9a44aa1741310bac4fa9b254f1aec65782b61505974eae1a61ed390fd",
"md5": "a56aa647b71092891d8f932e2726c227",
"sha256": "6935883f055a040a41d6af8f8597044de570c8229361886727d9b6b85bb18788"
},
"downloads": -1,
"filename": "thinc_apple_ops-0.1.4-cp311-cp311-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "a56aa647b71092891d8f932e2726c227",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.7",
"size": 161630,
"upload_time": "2023-09-22T14:50:06",
"upload_time_iso_8601": "2023-09-22T14:50:06.518799Z",
"url": "https://files.pythonhosted.org/packages/97/df/f0e9a44aa1741310bac4fa9b254f1aec65782b61505974eae1a61ed390fd/thinc_apple_ops-0.1.4-cp311-cp311-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fc5dda84b107dc3f67713b58efbc323001f0c391e65be28529eceae773f4435a",
"md5": "bc9768df135d633804ba5b985011599e",
"sha256": "dd098f13733e11f74444e6bd882068c35d7e9d15f40535526e420eb7a2ef56c9"
},
"downloads": -1,
"filename": "thinc_apple_ops-0.1.4-cp311-cp311-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "bc9768df135d633804ba5b985011599e",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.7",
"size": 154715,
"upload_time": "2023-09-22T14:50:07",
"upload_time_iso_8601": "2023-09-22T14:50:07.836555Z",
"url": "https://files.pythonhosted.org/packages/fc/5d/da84b107dc3f67713b58efbc323001f0c391e65be28529eceae773f4435a/thinc_apple_ops-0.1.4-cp311-cp311-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d3a362faa090ae7a5648be8850317b758353ee84bf000f9f5c1c75a00d617beb",
"md5": "4ba5049baa03227a8895ee509d2d60a8",
"sha256": "16cf0bd28a640177c62b1652915e225aa04458682bf698d9a51351654d206907"
},
"downloads": -1,
"filename": "thinc_apple_ops-0.1.4-cp312-cp312-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "4ba5049baa03227a8895ee509d2d60a8",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.7",
"size": 162812,
"upload_time": "2023-09-22T14:50:09",
"upload_time_iso_8601": "2023-09-22T14:50:09.439181Z",
"url": "https://files.pythonhosted.org/packages/d3/a3/62faa090ae7a5648be8850317b758353ee84bf000f9f5c1c75a00d617beb/thinc_apple_ops-0.1.4-cp312-cp312-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4368ce186496eef8a07ce1702eb20ebf95345fd9df4c57d35663f31ff0805853",
"md5": "2c009c01ff50044949e3e3fa658578f6",
"sha256": "30ebe3deecf0fb156b74ba4f600d52ec3fa96fb6efb8a00ca14d01e712a85cc4"
},
"downloads": -1,
"filename": "thinc_apple_ops-0.1.4-cp312-cp312-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "2c009c01ff50044949e3e3fa658578f6",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.7",
"size": 155753,
"upload_time": "2023-09-22T14:50:11",
"upload_time_iso_8601": "2023-09-22T14:50:11.076648Z",
"url": "https://files.pythonhosted.org/packages/43/68/ce186496eef8a07ce1702eb20ebf95345fd9df4c57d35663f31ff0805853/thinc_apple_ops-0.1.4-cp312-cp312-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "296c9802c46d1cfc10f498d84518de951a67c259d42d05c7e4e7455b9c9e02fa",
"md5": "9c067602da9a30918e35111df2b6b3c3",
"sha256": "fa9c53717b00f49de091046100e69c03556e209902c07eb50d0d4bbd26ba0f5d"
},
"downloads": -1,
"filename": "thinc_apple_ops-0.1.4-cp37-cp37m-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "9c067602da9a30918e35111df2b6b3c3",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.7",
"size": 162411,
"upload_time": "2023-09-22T14:50:12",
"upload_time_iso_8601": "2023-09-22T14:50:12.489324Z",
"url": "https://files.pythonhosted.org/packages/29/6c/9802c46d1cfc10f498d84518de951a67c259d42d05c7e4e7455b9c9e02fa/thinc_apple_ops-0.1.4-cp37-cp37m-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3eb6a3c6efa23b1dc5468cc10b2ec3451ddef16ded59837fdf49fb2e4f558846",
"md5": "bac3c5761d64bbbbb8554681b7c2e4e9",
"sha256": "9d7864bbee3e13a6c91ec59634810378d2dd11b808380ee40c11b3eb11aae1c6"
},
"downloads": -1,
"filename": "thinc_apple_ops-0.1.4-cp38-cp38-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "bac3c5761d64bbbbb8554681b7c2e4e9",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.7",
"size": 161782,
"upload_time": "2023-09-22T14:50:13",
"upload_time_iso_8601": "2023-09-22T14:50:13.867466Z",
"url": "https://files.pythonhosted.org/packages/3e/b6/a3c6efa23b1dc5468cc10b2ec3451ddef16ded59837fdf49fb2e4f558846/thinc_apple_ops-0.1.4-cp38-cp38-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a395840bf38872860836fba5c8ee0ba65835c0384a8a4fb535ac59e833ced48a",
"md5": "299a31f0d8c98c2b964f8aa5a643823c",
"sha256": "f203451c5c0dead351cd7fdc495afab6f7b6f40e27651e9f1d5f8839be127210"
},
"downloads": -1,
"filename": "thinc_apple_ops-0.1.4-cp38-cp38-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "299a31f0d8c98c2b964f8aa5a643823c",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.7",
"size": 155329,
"upload_time": "2023-09-22T14:50:15",
"upload_time_iso_8601": "2023-09-22T14:50:15.488121Z",
"url": "https://files.pythonhosted.org/packages/a3/95/840bf38872860836fba5c8ee0ba65835c0384a8a4fb535ac59e833ced48a/thinc_apple_ops-0.1.4-cp38-cp38-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e37eabf5eb5eb8108dc1345dd30beb3debeaed876fc8496d7483850c880de557",
"md5": "a8ad4dc8ff6004d15641cbe399aacccf",
"sha256": "31533f28e1af497edb246afa88cd7287ff49a365a1a21dfab2c6d2b19aea3990"
},
"downloads": -1,
"filename": "thinc_apple_ops-0.1.4-cp39-cp39-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "a8ad4dc8ff6004d15641cbe399aacccf",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.7",
"size": 162221,
"upload_time": "2023-09-22T14:50:16",
"upload_time_iso_8601": "2023-09-22T14:50:16.683525Z",
"url": "https://files.pythonhosted.org/packages/e3/7e/abf5eb5eb8108dc1345dd30beb3debeaed876fc8496d7483850c880de557/thinc_apple_ops-0.1.4-cp39-cp39-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3e1e15cfb9e470b6de158d20b195cba515e4513f72aa6ab054c5195055688fca",
"md5": "fc3a5f094be6e941e28704a58f99d243",
"sha256": "60cd002ceb2406991dda5adedc79e4494e19ba45a72d01c53124ecc86a00861f"
},
"downloads": -1,
"filename": "thinc_apple_ops-0.1.4-cp39-cp39-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "fc3a5f094be6e941e28704a58f99d243",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.7",
"size": 155330,
"upload_time": "2023-09-22T14:50:17",
"upload_time_iso_8601": "2023-09-22T14:50:17.825096Z",
"url": "https://files.pythonhosted.org/packages/3e/1e/15cfb9e470b6de158d20b195cba515e4513f72aa6ab054c5195055688fca/thinc_apple_ops-0.1.4-cp39-cp39-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ac9bbd3bc0bda5696adf82192cf6dfbebcdc99eaa83c7fda866579e489f86bde",
"md5": "c1371eb2c9213ca64c3f629bbafd81ad",
"sha256": "992dfc4805ab964131c9d9c71bcc4330becabe41916665ec424c035311347d05"
},
"downloads": -1,
"filename": "thinc_apple_ops-0.1.4.tar.gz",
"has_sig": false,
"md5_digest": "c1371eb2c9213ca64c3f629bbafd81ad",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 62013,
"upload_time": "2023-09-22T14:50:19",
"upload_time_iso_8601": "2023-09-22T14:50:19.069135Z",
"url": "https://files.pythonhosted.org/packages/ac/9b/bd3bc0bda5696adf82192cf6dfbebcdc99eaa83c7fda866579e489f86bde/thinc_apple_ops-0.1.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-22 14:50:19",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "explosion",
"github_project": "thinc-apple-ops",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "cython",
"specs": [
[
">=",
"0.25"
]
]
},
{
"name": "numpy",
"specs": [
[
">=",
"1.21.0"
]
]
},
{
"name": "thinc",
"specs": [
[
">=",
"8.1.0"
],
[
"<",
"9.1.0"
]
]
},
{
"name": "pytest",
"specs": [
[
">=",
"5.2.0"
]
]
}
],
"lcname": "thinc-apple-ops"
}
JSON
