PyProbables
===========
.. image:: https://img.shields.io/badge/license-MIT-blue.svg
:target: https://opensource.org/licenses/MIT/
:alt: License
.. image:: https://img.shields.io/github/release/barrust/pyprobables.svg
:target: https://github.com/barrust/pyprobables/releases
:alt: GitHub release
.. image:: https://github.com/barrust/pyprobables/workflows/Python%20package/badge.svg
:target: https://github.com/barrust/pyprobables/actions?query=workflow%3A%22Python+package%22
:alt: Build Status
.. image:: https://codecov.io/gh/barrust/pyprobables/branch/master/graph/badge.svg?token=OdETiNgz9k
:target: https://codecov.io/gh/barrust/pyprobables
:alt: Test Coverage
.. image:: https://readthedocs.org/projects/pyprobables/badge/?version=latest
:target: http://pyprobables.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
.. image:: https://badge.fury.io/py/pyprobables.svg
:target: https://pypi.org/project/pyprobables/
:alt: Pypi Release
.. image:: https://pepy.tech/badge/pyprobables
:target: https://pepy.tech/project/pyprobables
:alt: Downloads
**pyprobables** is a pure-python library for probabilistic data structures.
The goal is to provide the developer with a pure-python implementation of
common probabilistic data-structures to use in their work.
To achieve better raw performance, it is recommended supplying an alternative
hashing algorithm that has been compiled in C. This could include using the
md5 and sha512 algorithms provided or installing a third party package and
writing your own hashing strategy. Some options include the murmur hash
`mmh3 <https://github.com/hajimes/mmh3>`__ or those from the
`pyhash <https://github.com/flier/pyfasthash>`__ library. Each data object in
**pyprobables** makes it easy to pass in a custom hashing function.
Read more about how to use `Supplying a pre-defined, alternative hashing strategies`_
or `Defining hashing function using the provided decorators`_.
Installation
------------------
Pip Installation:
::
$ pip install pyprobables
To install from source:
To install `pyprobables`, simply clone the `repository on GitHub
<https://github.com/barrust/pyprobables>`__, then run from the folder:
::
$ python setup.py install
`pyprobables` supports python 3.6 - 3.11+
For *python 2.7* support, install `release 0.3.2 <https://github.com/barrust/pyprobables/releases/tag/v0.3.2>`__
::
$ pip install pyprobables==0.3.2
API Documentation
---------------------
The documentation of is hosted on
`readthedocs.io <http://pyprobables.readthedocs.io/en/latest/code.html#api>`__
You can build the documentation locally by running:
::
$ pip install sphinx
$ cd docs/
$ make html
Automated Tests
------------------
To run automated tests, one must simply run the following command from the
downloaded folder:
::
$ python setup.py test
Quickstart
------------------
Import pyprobables and setup a Bloom Filter
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: python
from probables import BloomFilter
blm = BloomFilter(est_elements=1000, false_positive_rate=0.05)
blm.add('google.com')
blm.check('facebook.com') # should return False
blm.check('google.com') # should return True
Import pyprobables and setup a Count-Min Sketch
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: python
from probables import CountMinSketch
cms = CountMinSketch(width=1000, depth=5)
cms.add('google.com') # should return 1
cms.add('facebook.com', 25) # insert 25 at once; should return 25
Import pyprobables and setup a Cuckoo Filter
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: python
from probables import CuckooFilter
cko = CuckooFilter(capacity=100, max_swaps=10)
cko.add('google.com')
cko.check('facebook.com') # should return False
cko.check('google.com') # should return True
Import pyprobables and setup a Quotient Filter
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: python
from probables import QuotientFilter
qf = QuotientFilter(quotient=24)
qf.add('google.com')
qf.check('facebook.com') # should return False
qf.check('google.com') # should return True
Supplying a pre-defined, alternative hashing strategies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: python
from probables import BloomFilter
from probables.hashes import default_sha256
blm = BloomFilter(est_elements=1000, false_positive_rate=0.05,
hash_function=default_sha256)
blm.add('google.com')
blm.check('facebook.com') # should return False
blm.check('google.com') # should return True
.. _use-custom-hashing-strategies:
Defining hashing function using the provided decorators
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: python
import mmh3 # murmur hash 3 implementation (pip install mmh3)
from probables.hashes import hash_with_depth_bytes
from probables import BloomFilter
@hash_with_depth_bytes
def my_hash(key, depth):
return mmh3.hash_bytes(key, seed=depth)
blm = BloomFilter(est_elements=1000, false_positive_rate=0.05, hash_function=my_hash)
.. code:: python
import hashlib
from probables.hashes import hash_with_depth_int
from probables.constants import UINT64_T_MAX
from probables import BloomFilter
@hash_with_depth_int
def my_hash(key, seed=0, encoding="utf-8"):
max64mod = UINT64_T_MAX + 1
val = int(hashlib.sha512(key.encode(encoding)).hexdigest(), 16)
val += seed # not a good example, but uses the seed value
return val % max64mod
blm = BloomFilter(est_elements=1000, false_positive_rate=0.05, hash_function=my_hash)
See the `API documentation <http://pyprobables.readthedocs.io/en/latest/code.html#api>`__
for other data structures available and the
`quickstart page <http://pyprobables.readthedocs.io/en/latest/quickstart.html#quickstart>`__
for more examples!
Changelog
------------------
Please see the `changelog
<https://github.com/barrust/pyprobables/blob/master/CHANGELOG.md>`__ for a list
of all changes.
Backward Compatible Changes
---------------------------
If you are using previously exported probablistic data structures (v0.4.1 or below)
and used the default hashing strategy, you will want to use the following code
to mimic the original default hashing algorithm.
.. code:: python
from probables import BloomFilter
from probables.hashes import hash_with_depth_int
@hash_with_depth_int
def old_fnv1a(key, depth=1):
return tmp_fnv_1a(key)
def tmp_fnv_1a(key):
max64mod = UINT64_T_MAX + 1
hval = 14695981039346656073
fnv_64_prime = 1099511628211
tmp = map(ord, key)
for t_str in tmp:
hval ^= t_str
hval *= fnv_64_prime
hval %= max64mod
return hval
blm = BloomFilter(filpath="old-file-path.blm", hash_function=old_fnv1a)
Raw data
{
"_id": null,
"home_page": "",
"name": "pyprobables",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "python,probabilistic,data-structure,bloom,filter,count-min,sketch,bloom-filter,count-min-sketch,cuckoo-filter",
"author": "",
"author_email": "Tyler Barrus <barrust@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/1d/fa/3dfaaa1ff7ce5867a890fd3d86dfb24ba7da1f5c0fdd03d7e8d86a1afa9a/pyprobables-0.6.0.tar.gz",
"platform": null,
"description": "PyProbables\n===========\n\n.. image:: https://img.shields.io/badge/license-MIT-blue.svg\n :target: https://opensource.org/licenses/MIT/\n :alt: License\n.. image:: https://img.shields.io/github/release/barrust/pyprobables.svg\n :target: https://github.com/barrust/pyprobables/releases\n :alt: GitHub release\n.. image:: https://github.com/barrust/pyprobables/workflows/Python%20package/badge.svg\n :target: https://github.com/barrust/pyprobables/actions?query=workflow%3A%22Python+package%22\n :alt: Build Status\n.. image:: https://codecov.io/gh/barrust/pyprobables/branch/master/graph/badge.svg?token=OdETiNgz9k\n :target: https://codecov.io/gh/barrust/pyprobables\n :alt: Test Coverage\n.. image:: https://readthedocs.org/projects/pyprobables/badge/?version=latest\n :target: http://pyprobables.readthedocs.io/en/latest/?badge=latest\n :alt: Documentation Status\n.. image:: https://badge.fury.io/py/pyprobables.svg\n :target: https://pypi.org/project/pyprobables/\n :alt: Pypi Release\n.. image:: https://pepy.tech/badge/pyprobables\n :target: https://pepy.tech/project/pyprobables\n :alt: Downloads\n\n**pyprobables** is a pure-python library for probabilistic data structures.\nThe goal is to provide the developer with a pure-python implementation of\ncommon probabilistic data-structures to use in their work.\n\nTo achieve better raw performance, it is recommended supplying an alternative\nhashing algorithm that has been compiled in C. This could include using the\nmd5 and sha512 algorithms provided or installing a third party package and\nwriting your own hashing strategy. Some options include the murmur hash\n`mmh3 <https://github.com/hajimes/mmh3>`__ or those from the\n`pyhash <https://github.com/flier/pyfasthash>`__ library. Each data object in\n**pyprobables** makes it easy to pass in a custom hashing function.\n\nRead more about how to use `Supplying a pre-defined, alternative hashing strategies`_\nor `Defining hashing function using the provided decorators`_.\n\nInstallation\n------------------\n\nPip Installation:\n\n::\n\n $ pip install pyprobables\n\nTo install from source:\n\nTo install `pyprobables`, simply clone the `repository on GitHub\n<https://github.com/barrust/pyprobables>`__, then run from the folder:\n\n::\n\n $ python setup.py install\n\n`pyprobables` supports python 3.6 - 3.11+\n\nFor *python 2.7* support, install `release 0.3.2 <https://github.com/barrust/pyprobables/releases/tag/v0.3.2>`__\n\n::\n\n $ pip install pyprobables==0.3.2\n\n\nAPI Documentation\n---------------------\n\nThe documentation of is hosted on\n`readthedocs.io <http://pyprobables.readthedocs.io/en/latest/code.html#api>`__\n\nYou can build the documentation locally by running:\n\n::\n\n $ pip install sphinx\n $ cd docs/\n $ make html\n\n\n\nAutomated Tests\n------------------\n\nTo run automated tests, one must simply run the following command from the\ndownloaded folder:\n\n::\n\n $ python setup.py test\n\n\n\nQuickstart\n------------------\n\nImport pyprobables and setup a Bloom Filter\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code:: python\n\n from probables import BloomFilter\n blm = BloomFilter(est_elements=1000, false_positive_rate=0.05)\n blm.add('google.com')\n blm.check('facebook.com') # should return False\n blm.check('google.com') # should return True\n\n\nImport pyprobables and setup a Count-Min Sketch\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code:: python\n\n from probables import CountMinSketch\n cms = CountMinSketch(width=1000, depth=5)\n cms.add('google.com') # should return 1\n cms.add('facebook.com', 25) # insert 25 at once; should return 25\n\n\nImport pyprobables and setup a Cuckoo Filter\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code:: python\n\n from probables import CuckooFilter\n cko = CuckooFilter(capacity=100, max_swaps=10)\n cko.add('google.com')\n cko.check('facebook.com') # should return False\n cko.check('google.com') # should return True\n\n\nImport pyprobables and setup a Quotient Filter\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code:: python\n\n from probables import QuotientFilter\n qf = QuotientFilter(quotient=24)\n qf.add('google.com')\n qf.check('facebook.com') # should return False\n qf.check('google.com') # should return True\n\n\nSupplying a pre-defined, alternative hashing strategies\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code:: python\n\n from probables import BloomFilter\n from probables.hashes import default_sha256\n blm = BloomFilter(est_elements=1000, false_positive_rate=0.05,\n hash_function=default_sha256)\n blm.add('google.com')\n blm.check('facebook.com') # should return False\n blm.check('google.com') # should return True\n\n\n.. _use-custom-hashing-strategies:\n\nDefining hashing function using the provided decorators\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code:: python\n\n import mmh3 # murmur hash 3 implementation (pip install mmh3)\n from probables.hashes import hash_with_depth_bytes\n from probables import BloomFilter\n\n @hash_with_depth_bytes\n def my_hash(key, depth):\n return mmh3.hash_bytes(key, seed=depth)\n\n blm = BloomFilter(est_elements=1000, false_positive_rate=0.05, hash_function=my_hash)\n\n.. code:: python\n\n import hashlib\n from probables.hashes import hash_with_depth_int\n from probables.constants import UINT64_T_MAX\n from probables import BloomFilter\n\n @hash_with_depth_int\n def my_hash(key, seed=0, encoding=\"utf-8\"):\n max64mod = UINT64_T_MAX + 1\n val = int(hashlib.sha512(key.encode(encoding)).hexdigest(), 16)\n val += seed # not a good example, but uses the seed value\n return val % max64mod\n\n blm = BloomFilter(est_elements=1000, false_positive_rate=0.05, hash_function=my_hash)\n\n\nSee the `API documentation <http://pyprobables.readthedocs.io/en/latest/code.html#api>`__\nfor other data structures available and the\n`quickstart page <http://pyprobables.readthedocs.io/en/latest/quickstart.html#quickstart>`__\nfor more examples!\n\n\nChangelog\n------------------\n\nPlease see the `changelog\n<https://github.com/barrust/pyprobables/blob/master/CHANGELOG.md>`__ for a list\nof all changes.\n\n\nBackward Compatible Changes\n---------------------------\n\nIf you are using previously exported probablistic data structures (v0.4.1 or below)\nand used the default hashing strategy, you will want to use the following code\nto mimic the original default hashing algorithm.\n\n.. code:: python\n\n from probables import BloomFilter\n from probables.hashes import hash_with_depth_int\n\n @hash_with_depth_int\n def old_fnv1a(key, depth=1):\n return tmp_fnv_1a(key)\n\n def tmp_fnv_1a(key):\n max64mod = UINT64_T_MAX + 1\n hval = 14695981039346656073\n fnv_64_prime = 1099511628211\n tmp = map(ord, key)\n for t_str in tmp:\n hval ^= t_str\n hval *= fnv_64_prime\n hval %= max64mod\n return hval\n\n blm = BloomFilter(filpath=\"old-file-path.blm\", hash_function=old_fnv1a)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Probabilistic data structures in python",
"version": "0.6.0",
"project_urls": {
"Bug-tracker": "https://github.com/barrust/pyprobables/issues",
"Documentation": "https://pyprobables.readthedocs.io/",
"Homepage": "https://github.com/barrust/pyprobables"
},
"split_keywords": [
"python",
"probabilistic",
"data-structure",
"bloom",
"filter",
"count-min",
"sketch",
"bloom-filter",
"count-min-sketch",
"cuckoo-filter"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2548985c0f0a8a6d62a7d2591e49732254c13df1ea7d33a508a3e528ebe9a666",
"md5": "66eba07abd7518f6c273c45d70294399",
"sha256": "85a23655e2d5e87a99e01003be8b48e9dfcc0c45e65c02f84f777b99235347db"
},
"downloads": -1,
"filename": "pyprobables-0.6.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "66eba07abd7518f6c273c45d70294399",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 40035,
"upload_time": "2024-01-12T22:39:57",
"upload_time_iso_8601": "2024-01-12T22:39:57.353694Z",
"url": "https://files.pythonhosted.org/packages/25/48/985c0f0a8a6d62a7d2591e49732254c13df1ea7d33a508a3e528ebe9a666/pyprobables-0.6.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1dfa3dfaaa1ff7ce5867a890fd3d86dfb24ba7da1f5c0fdd03d7e8d86a1afa9a",
"md5": "ac704a95b92585410ec1807f5872e86f",
"sha256": "a4e72bdb4d3513121b33377728c9eafd2ae8495d5201d6a90abc3d52d9a17901"
},
"downloads": -1,
"filename": "pyprobables-0.6.0.tar.gz",
"has_sig": false,
"md5_digest": "ac704a95b92585410ec1807f5872e86f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 33638,
"upload_time": "2024-01-12T22:39:59",
"upload_time_iso_8601": "2024-01-12T22:39:59.348608Z",
"url": "https://files.pythonhosted.org/packages/1d/fa/3dfaaa1ff7ce5867a890fd3d86dfb24ba7da1f5c0fdd03d7e8d86a1afa9a/pyprobables-0.6.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-12 22:39:59",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "barrust",
"github_project": "pyprobables",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pyprobables"
}
JSON
