# PYMECompress
Compression for photon-noise limited images which keeps losses within the Poisson noise envelope
PYMECompress consists of four parts:
- a fork of the Basic Compression Library originally by Marcus Geelnard,
modified to include a heavily optimized huffman coder (BCL license is avalable under pymecompress/bcl/doc/manual.pdf and would appear to be BSD compatible)
- a fast, AVX optimized, quantizer to perform "within noise level" quantization of photon-limited images
- a python wrapper of the above. Note that at this point, only huffman coding and quantization are exposed to python
- numcodecs codecs (experimental) to permit simple usage with other IO packages - e.g. Zarr
Together they offer a single core throughput of ~500 -600MB/s
## Installation
### conda
Prebuilt binaries of PYMEcompress are available as a conda package (*pymecompress*) on the *david_baddeley* conda channel for python 2.7, 3.6 & 3.7, and installable via:
conda install -c david_baddeley pymecompress
### source
If you want to modify/contribute to the package you will have to build from source.
Because we use gcc compiler extensions for avx opcodes, we must use gcc/clang for compilation, regardless of platform.
On OSX / linux, a standard `python setup.py install` or `python setup.py develop` should work.
On Windows, you need to install mingw and run the build step first so that you can pass the compiler flag to `python setup.py build` - i.e. :
python setup.py build --compiler=mingw32
python setup.py install
A suitable environment for building pymecompress can be created using the following conda command `conda create -n <name> python=x.x numpy cython libpython m2w64-toolchain`
### pip
Installation via pip is also available:
pip install pymecompress
although binary wheels are not available for all platforms so you may need to set up a build environment (gcc/mingw, as described for source installation) first.
## Usage
### numcodecs codecs
```python
import numpy as np
from pymecompress import codecs
# vanilla huffman coding (lossless). NB - input buffer must be bytes/uint8
huff = codecs.Huffman()
d = np.ones(1000)
assert np.allclose(huff.decode(huff.encode(d.view('uint8'))).view(d.dtype), d)
# with quantisation NB: input data type MUST be uint16
huffq = codecs.HuffmanQuant16(offset=0, scale=1.0)
ds = np.linspace(1,2**15).astype('uint16')
assert np.all((huffq.decode(huffq.encode(ds)) - ds.astype('f')) < np.sqrt(ds))
```
### As a Zarr compression filter
VERY EXPERIMENTAL!
This is not yet well tested, but should work as described in https://zarr.readthedocs.io/en/stable/api/codecs.html. In brief ...
```python
import zarr
from pymecompress import codecs
z = zarr.zeros(1000000, dtype='uint16', compressor=codecs.HuffmanQuant16(offset=0, scale=1))
```
**NB** To be able to read/open files saved using the pymecompress codecs you will probably need to run `from pymecompress import codecs`
to register the codecs with `numcodecs` before trying to open the file.
### Directly calling functions
As you need to supply the original size to the decompression function, these are most suitable when putting the compressed data in an external wrapper e.g. PYMEs' PZFFormat which keeps track of the original data dimensions and dtype (we save a couple of bytes and a seek over using the codec versions above).
```python
import numpy as np
import pymecompress
data = np.linspace(1,2**15).astype('uint16')
nbytes = data.nbytes
c = pymecompress.HuffmanCompressQuant(data, quantizationOffset, quantizationScale).to_string()
decompressed = pymecompress.HuffmanDecompress(np.fromstring(c, 'u1'), nbytes)
dequantized = (quantisationScale*decompressed)**2 + quantizationOffset
```
Raw data
{
"_id": null,
"home_page": "https://github.com/python-microscopy/pymecompress",
"name": "pymecompress",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "",
"author": "David Baddeley",
"author_email": "David Baddeley <d.baddeley@auckland.ac.nz>",
"download_url": "https://files.pythonhosted.org/packages/af/4f/943f13b22fbd392dc5fd2052ca45d99ddbe2662823dd0e6456b5322f7331/pymecompress-0.3.6.tar.gz",
"platform": null,
"description": "# PYMECompress\n\n\n\n\n\nCompression for photon-noise limited images which keeps losses within the Poisson noise envelope\n\nPYMECompress consists of four parts: \n\n- a fork of the Basic Compression Library originally by Marcus Geelnard, \nmodified to include a heavily optimized huffman coder (BCL license is avalable under pymecompress/bcl/doc/manual.pdf and would appear to be BSD compatible)\n\n- a fast, AVX optimized, quantizer to perform \"within noise level\" quantization of photon-limited images\n\n- a python wrapper of the above. Note that at this point, only huffman coding and quantization are exposed to python\n\n- numcodecs codecs (experimental) to permit simple usage with other IO packages - e.g. Zarr\n\nTogether they offer a single core throughput of ~500 -600MB/s\n\n\n## Installation\n\n### conda\n\nPrebuilt binaries of PYMEcompress are available as a conda package (*pymecompress*) on the *david_baddeley* conda channel for python 2.7, 3.6 & 3.7, and installable via:\n\n conda install -c david_baddeley pymecompress\n\n### source\n\nIf you want to modify/contribute to the package you will have to build from source.\n\nBecause we use gcc compiler extensions for avx opcodes, we must use gcc/clang for compilation, regardless of platform.\n\nOn OSX / linux, a standard `python setup.py install` or `python setup.py develop` should work.\n\nOn Windows, you need to install mingw and run the build step first so that you can pass the compiler flag to `python setup.py build` - i.e. :\n\n python setup.py build --compiler=mingw32\n python setup.py install\n\n\nA suitable environment for building pymecompress can be created using the following conda command `conda create -n <name> python=x.x numpy cython libpython m2w64-toolchain`\n\n### pip\n\nInstallation via pip is also available:\n\n pip install pymecompress\n\nalthough binary wheels are not available for all platforms so you may need to set up a build environment (gcc/mingw, as described for source installation) first.\n\n## Usage\n\n### numcodecs codecs\n\n```python\nimport numpy as np\nfrom pymecompress import codecs\n\n# vanilla huffman coding (lossless). NB - input buffer must be bytes/uint8\nhuff = codecs.Huffman()\nd = np.ones(1000)\nassert np.allclose(huff.decode(huff.encode(d.view('uint8'))).view(d.dtype), d)\n\n\n# with quantisation NB: input data type MUST be uint16\nhuffq = codecs.HuffmanQuant16(offset=0, scale=1.0)\nds = np.linspace(1,2**15).astype('uint16')\n\nassert np.all((huffq.decode(huffq.encode(ds)) - ds.astype('f')) < np.sqrt(ds))\n\n```\n\n### As a Zarr compression filter\n\nVERY EXPERIMENTAL! \n\nThis is not yet well tested, but should work as described in https://zarr.readthedocs.io/en/stable/api/codecs.html. In brief ...\n\n```python\nimport zarr\nfrom pymecompress import codecs\nz = zarr.zeros(1000000, dtype='uint16', compressor=codecs.HuffmanQuant16(offset=0, scale=1))\n\n```\n\n**NB** To be able to read/open files saved using the pymecompress codecs you will probably need to run `from pymecompress import codecs`\nto register the codecs with `numcodecs` before trying to open the file.\n\n### Directly calling functions\n\nAs you need to supply the original size to the decompression function, these are most suitable when putting the compressed data in an external wrapper e.g. PYMEs' PZFFormat which keeps track of the original data dimensions and dtype (we save a couple of bytes and a seek over using the codec versions above). \n\n```python\n\nimport numpy as np\nimport pymecompress\n\ndata = np.linspace(1,2**15).astype('uint16')\n\nnbytes = data.nbytes\nc = pymecompress.HuffmanCompressQuant(data, quantizationOffset, quantizationScale).to_string()\ndecompressed = pymecompress.HuffmanDecompress(np.fromstring(c, 'u1'), nbytes)\ndequantized = (quantisationScale*decompressed)**2 + quantizationOffset\n\n```\n",
"bugtrack_url": null,
"license": "BSD",
"summary": "Compression for photon-noise limited images which keeps losses within the Poisson noise envelope",
"version": "0.3.6",
"project_urls": {
"Homepage": "https://github.com/python-microscopy/pymecompress"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1d12d44f4e54e8660025cac12e83811a3f52cd9bc605f4ccdf88f038536d8bd9",
"md5": "68a5d86128f48aa504a8dbfff86720f2",
"sha256": "9522737bb04f7ff00e946eb22b65d34fe0c86061949d92d79668df82517a30ce"
},
"downloads": -1,
"filename": "pymecompress-0.3.6-cp311-cp311-macosx_12_0_x86_64.whl",
"has_sig": false,
"md5_digest": "68a5d86128f48aa504a8dbfff86720f2",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.6",
"size": 93146,
"upload_time": "2023-08-13T23:43:33",
"upload_time_iso_8601": "2023-08-13T23:43:33.159962Z",
"url": "https://files.pythonhosted.org/packages/1d/12/d44f4e54e8660025cac12e83811a3f52cd9bc605f4ccdf88f038536d8bd9/pymecompress-0.3.6-cp311-cp311-macosx_12_0_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fa496a0a58bf13fa237d148b36e96b66953e6ba03144ae5acb83979646a081ae",
"md5": "ff36b09ada952e4a49dc10c55774e5a4",
"sha256": "f22505e38c46ef51c18094a128452bd8bb2ccdebc687c7b9ce90345cee4889ba"
},
"downloads": -1,
"filename": "pymecompress-0.3.6-cp37-cp37m-win_amd64.whl",
"has_sig": false,
"md5_digest": "ff36b09ada952e4a49dc10c55774e5a4",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.6",
"size": 348991,
"upload_time": "2023-08-13T23:45:44",
"upload_time_iso_8601": "2023-08-13T23:45:44.793538Z",
"url": "https://files.pythonhosted.org/packages/fa/49/6a0a58bf13fa237d148b36e96b66953e6ba03144ae5acb83979646a081ae/pymecompress-0.3.6-cp37-cp37m-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "cdb12515aed765238db841e9e135480dbf5adce27c35dabcce1548a79634580f",
"md5": "c07f8450440f932ed27703dabc617d42",
"sha256": "fd1f2a117e5826ab8056b9c804c7d3d0978646e008ec5ca59090f61e1b4726ec"
},
"downloads": -1,
"filename": "pymecompress-0.3.6-cp38-cp38-win_amd64.whl",
"has_sig": false,
"md5_digest": "c07f8450440f932ed27703dabc617d42",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.6",
"size": 414580,
"upload_time": "2023-08-13T23:45:59",
"upload_time_iso_8601": "2023-08-13T23:45:59.485708Z",
"url": "https://files.pythonhosted.org/packages/cd/b1/2515aed765238db841e9e135480dbf5adce27c35dabcce1548a79634580f/pymecompress-0.3.6-cp38-cp38-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2909f2991948bd1568569f3cd3f73fdd46aaea03dfb630c818da6dc02a6744c2",
"md5": "8ae947e1eaf0e09a26f647462705feb0",
"sha256": "3057288db6dea9c2c543fe03aa9462ac55c3934179bf26fd99a8bc9a6312e554"
},
"downloads": -1,
"filename": "pymecompress-0.3.6-cp39-cp39-win_amd64.whl",
"has_sig": false,
"md5_digest": "8ae947e1eaf0e09a26f647462705feb0",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.6",
"size": 399418,
"upload_time": "2023-08-13T23:45:54",
"upload_time_iso_8601": "2023-08-13T23:45:54.077947Z",
"url": "https://files.pythonhosted.org/packages/29/09/f2991948bd1568569f3cd3f73fdd46aaea03dfb630c818da6dc02a6744c2/pymecompress-0.3.6-cp39-cp39-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "af4f943f13b22fbd392dc5fd2052ca45d99ddbe2662823dd0e6456b5322f7331",
"md5": "0be66682fb138fe7f94c570230b3f715",
"sha256": "cc13a7ed0b19edf3367ab2de5ef7fb3b28791c91fae7409d8a5fe70ae7596de7"
},
"downloads": -1,
"filename": "pymecompress-0.3.6.tar.gz",
"has_sig": false,
"md5_digest": "0be66682fb138fe7f94c570230b3f715",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 156803,
"upload_time": "2023-08-13T23:42:12",
"upload_time_iso_8601": "2023-08-13T23:42:12.088491Z",
"url": "https://files.pythonhosted.org/packages/af/4f/943f13b22fbd392dc5fd2052ca45d99ddbe2662823dd0e6456b5322f7331/pymecompress-0.3.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-13 23:42:12",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "python-microscopy",
"github_project": "pymecompress",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "numpy",
"specs": [
[
">=",
"1.11"
]
]
},
{
"name": "cython",
"specs": []
},
{
"name": "six",
"specs": []
}
],
"lcname": "pymecompress"
}
JSON
