# CompIntPy
A Python library for compressing integers with variable-length Elias compression algorithms ([universal codes](https://en.wikipedia.org/wiki/Universal_code_(data_compression))). The library was designed for compressing indices of weight matrices in decentralised machine learning. Hence, it supports multithreading for faster encoding of indices to speed up the distribution of the data to neighbouring nodes.
## Supported Algorithms
The library implements the Elias [gamma](https://en.wikipedia.org/wiki/Elias_gamma_coding), [delta](https://en.wikipedia.org/wiki/Elias_delta_coding), and [omega](https://en.wikipedia.org/wiki/Elias_omega_coding) algorithms. An example for `EliasGamma` is given below (Replace `Gamma` with your respective choices):
```
arr = np.array([0, -1, 2, -5, 8], dtype=np.int64)
eg = EliasGamma(offset=1, map_negative_numbers=True)
comp = eg.compress(arr)
uncomp = eg.decompress(output, arr.size, output_dtype=np.int64)
```
## Installation
For Linux the project is available via pip:
```
pip install compintpy
```
Or, install it from the GitHub repository:
```
git clone --recursive https://github.com/JeffWigger/compintpy.git
cd compintpy
pip install .
```
## Documentation
All three algorithms are available as Python objects: `comppy.elias.EliasGamma`, `comppy.elias.EliasDelta`, and `comppy.elias.EliasOmega`.
They are all subclasses of the follwing class:
```
class comppy.elias.Elias(offset: int = 0, map_negative_numbers: bool = False)
""" Abstract base class of all Elias compression implementations.""""
abstract compress(array: ndarray) → ndarray
""" Abstract method defining the compression method used by all Elias implementations.
Parameters:
array (np.ndarray) – Array to be compressed. The array must be of types Union[np.int64, np.uint64, np.int32, np.uint32].
"""
abstract decompress(array: ndarray, output_length: int, output_dtype: int64 | uint64 | int32 | uint32) → ndarray
"""Abstract method defining the decompression method used by all Elias implementations.
Parameters:
array (np.ndarray) – Numpy array containing data compressed with an Elias algorithm. The dtype must be np.uint8.
output_length (int) – Length of the decompressed output.
output_dtype (Union[np.int64, np.uint64, np.int32, np.uint32]) – Dtype of the output array.
""""
```
## Multi-threading
The compress function is parallelized with OpenMP. You can set the number of threads by setting the OMP_NUM_THREADS environment variable, e.g.,
export OMP_NUM_THREADS=10
## Bindings
This library binds the high-performance C++ code from our sister Project [ComIntC](https://github.com/JeffWigger/compintc).
Raw data
{
"_id": null,
"home_page": "https://github.com/JeffWigger/compintpy",
"name": "compintpy",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": "Jeffrey Wigger",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/01/de/bce1b781bdb1affdc28c8c5e6380f4ac9b4aae4ec258a121c7ff53fdce16/compintpy-0.0.4.tar.gz",
"platform": null,
"description": "# CompIntPy\n\n\nA Python library for compressing integers with variable-length Elias compression algorithms ([universal codes](https://en.wikipedia.org/wiki/Universal_code_(data_compression))). The library was designed for compressing indices of weight matrices in decentralised machine learning. Hence, it supports multithreading for faster encoding of indices to speed up the distribution of the data to neighbouring nodes.\n\n\n## Supported Algorithms\n\n\n\nThe library implements the Elias [gamma](https://en.wikipedia.org/wiki/Elias_gamma_coding), [delta](https://en.wikipedia.org/wiki/Elias_delta_coding), and [omega](https://en.wikipedia.org/wiki/Elias_omega_coding) algorithms. An example for `EliasGamma` is given below (Replace `Gamma` with your respective choices):\n\n\n```\narr = np.array([0, -1, 2, -5, 8], dtype=np.int64)\neg = EliasGamma(offset=1, map_negative_numbers=True)\ncomp = eg.compress(arr)\nuncomp = eg.decompress(output, arr.size, output_dtype=np.int64)\n```\n\n## Installation\n\nFor Linux the project is available via pip:\n```\npip install compintpy\n```\n\nOr, install it from the GitHub repository:\n```\ngit clone --recursive https://github.com/JeffWigger/compintpy.git\ncd compintpy\npip install .\n```\n\n## Documentation\nAll three algorithms are available as Python objects: `comppy.elias.EliasGamma`, `comppy.elias.EliasDelta`, and `comppy.elias.EliasOmega`.\nThey are all subclasses of the follwing class:\n```\nclass comppy.elias.Elias(offset: int = 0, map_negative_numbers: bool = False)\n \"\"\" Abstract base class of all Elias compression implementations.\"\"\"\"\n abstract compress(array: ndarray) \u2192 ndarray\n \"\"\" Abstract method defining the compression method used by all Elias implementations.\n Parameters:\n array (np.ndarray) \u2013 Array to be compressed. The array must be of types Union[np.int64, np.uint64, np.int32, np.uint32].\n \"\"\"\n abstract decompress(array: ndarray, output_length: int, output_dtype: int64 | uint64 | int32 | uint32) \u2192 ndarray\n \"\"\"Abstract method defining the decompression method used by all Elias implementations.\n Parameters:\n array (np.ndarray) \u2013 Numpy array containing data compressed with an Elias algorithm. The dtype must be np.uint8.\n output_length (int) \u2013 Length of the decompressed output.\n output_dtype (Union[np.int64, np.uint64, np.int32, np.uint32]) \u2013 Dtype of the output array.\n \"\"\"\"\n```\n\n\n## Multi-threading\nThe compress function is parallelized with OpenMP. You can set the number of threads by setting the OMP_NUM_THREADS environment variable, e.g.,\n\n\nexport OMP_NUM_THREADS=10\n\n\n## Bindings\n\n\nThis library binds the high-performance C++ code from our sister Project [ComIntC](https://github.com/JeffWigger/compintc).\n",
"bugtrack_url": null,
"license": "BSD-3-Clause",
"summary": "Python bindings for compintc: A variable length compression algorithm.",
"version": "0.0.4",
"project_urls": {
"Homepage": "https://github.com/JeffWigger/compintpy"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7c034bf43c3cdf20c37ef3fcca736524266258149d51b08019649ecbdf38dcbf",
"md5": "e207f57b6bf8ccd081472f83b71bf8b6",
"sha256": "721f326d247991134152d1ac2c2f9a817abb5948dca79e44165f04b85f2d6a5e"
},
"downloads": -1,
"filename": "compintpy-0.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "e207f57b6bf8ccd081472f83b71bf8b6",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.9",
"size": 187956,
"upload_time": "2024-11-10T21:05:26",
"upload_time_iso_8601": "2024-11-10T21:05:26.627570Z",
"url": "https://files.pythonhosted.org/packages/7c/03/4bf43c3cdf20c37ef3fcca736524266258149d51b08019649ecbdf38dcbf/compintpy-0.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "76141e42b73248f5eab8ad904f171379758eaaa34dc726b24c9fb1c7acc34a45",
"md5": "dbf8cc9cc1fe2a5602214b274cea1493",
"sha256": "00de8c0e7a2cf6158debb8c5ed2d9246ccc3d816cc83db2b02c7c17a9cb7ece3"
},
"downloads": -1,
"filename": "compintpy-0.0.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "dbf8cc9cc1fe2a5602214b274cea1493",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.9",
"size": 190332,
"upload_time": "2024-11-10T21:05:28",
"upload_time_iso_8601": "2024-11-10T21:05:28.476507Z",
"url": "https://files.pythonhosted.org/packages/76/14/1e42b73248f5eab8ad904f171379758eaaa34dc726b24c9fb1c7acc34a45/compintpy-0.0.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "235c21ec07a8516283b0caff8c8efbe1fbaaac7649ba2c611e08b3a801f0f7e0",
"md5": "eed84af685cbb126c177ac0192566842",
"sha256": "ed7a3e356ae8beaeac99f751df69229a0dc39c33743ba171e590cc1effe5b62c"
},
"downloads": -1,
"filename": "compintpy-0.0.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "eed84af685cbb126c177ac0192566842",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.9",
"size": 188967,
"upload_time": "2024-11-10T21:05:30",
"upload_time_iso_8601": "2024-11-10T21:05:30.224413Z",
"url": "https://files.pythonhosted.org/packages/23/5c/21ec07a8516283b0caff8c8efbe1fbaaac7649ba2c611e08b3a801f0f7e0/compintpy-0.0.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "dbc26cfb7e4db6419588825ae661477fcd21d858f4b0de10227389a07aad60a2",
"md5": "78035e708682ef2bd1024f80fc83ca5a",
"sha256": "26c987bfd52f3e4dbe90ba16b42c57271fdd3e93cd2b93cc9957671bdea9ec0b"
},
"downloads": -1,
"filename": "compintpy-0.0.4-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "78035e708682ef2bd1024f80fc83ca5a",
"packagetype": "bdist_wheel",
"python_version": "cp313",
"requires_python": ">=3.9",
"size": 190164,
"upload_time": "2024-11-10T21:05:31",
"upload_time_iso_8601": "2024-11-10T21:05:31.943503Z",
"url": "https://files.pythonhosted.org/packages/db/c2/6cfb7e4db6419588825ae661477fcd21d858f4b0de10227389a07aad60a2/compintpy-0.0.4-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b5db1acd336502e3e3ad173d807a4627343a93ee1b04161252617bd99e2204fa",
"md5": "b6e1030e186077d6e692e0b94c4aeb8c",
"sha256": "88788b2f141faad3ced1abe53cfcfae71518c6beb25d97c629d0cf694e8073dc"
},
"downloads": -1,
"filename": "compintpy-0.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "b6e1030e186077d6e692e0b94c4aeb8c",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.9",
"size": 188685,
"upload_time": "2024-11-10T21:05:33",
"upload_time_iso_8601": "2024-11-10T21:05:33.663323Z",
"url": "https://files.pythonhosted.org/packages/b5/db/1acd336502e3e3ad173d807a4627343a93ee1b04161252617bd99e2204fa/compintpy-0.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "01debce1b781bdb1affdc28c8c5e6380f4ac9b4aae4ec258a121c7ff53fdce16",
"md5": "6fac1530f55979ce9b1ce6036c5b01a6",
"sha256": "6c39bc4b01e381155bcc503c5f15d5077b2292504792ff7c1b1cfd3164cdc4ab"
},
"downloads": -1,
"filename": "compintpy-0.0.4.tar.gz",
"has_sig": false,
"md5_digest": "6fac1530f55979ce9b1ce6036c5b01a6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 259974,
"upload_time": "2024-11-10T21:05:35",
"upload_time_iso_8601": "2024-11-10T21:05:35.600685Z",
"url": "https://files.pythonhosted.org/packages/01/de/bce1b781bdb1affdc28c8c5e6380f4ac9b4aae4ec258a121c7ff53fdce16/compintpy-0.0.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-10 21:05:35",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "JeffWigger",
"github_project": "compintpy",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"appveyor": true,
"lcname": "compintpy"
}
JSON
