datasketch: Big Data Looks Small
================================
.. image:: https://github.com/ekzhu/datasketch/workflows/Python%20package/badge.svg
:target: https://github.com/ekzhu/datasketch/actions
.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.290602.svg
:target: https://doi.org/10.5281/zenodo.290602
datasketch gives you probabilistic data structures that can process and
search very large amount of data super fast, with little loss of
accuracy.
This package contains the following data sketches:
+-------------------------+-----------------------------------------------+
| Data Sketch | Usage |
+=========================+===============================================+
| `MinHash`_ | estimate Jaccard similarity and cardinality |
+-------------------------+-----------------------------------------------+
| `Weighted MinHash`_ | estimate weighted Jaccard similarity |
+-------------------------+-----------------------------------------------+
| `HyperLogLog`_ | estimate cardinality |
+-------------------------+-----------------------------------------------+
| `HyperLogLog++`_ | estimate cardinality |
+-------------------------+-----------------------------------------------+
The following indexes for data sketches are provided to support
sub-linear query time:
+---------------------------+-----------------------------+------------------------+
| Index | For Data Sketch | Supported Query Type |
+===========================+=============================+========================+
| `MinHash LSH`_ | MinHash, Weighted MinHash | Jaccard Threshold |
+---------------------------+-----------------------------+------------------------+
| `MinHash LSH Forest`_ | MinHash, Weighted MinHash | Jaccard Top-K |
+---------------------------+-----------------------------+------------------------+
| `MinHash LSH Ensemble`_ | MinHash | Containment Threshold |
+---------------------------+-----------------------------+------------------------+
| `HNSW`_ | Any | Custom Metric Top-K |
+---------------------------+-----------------------------+------------------------+
datasketch must be used with Python 3.7 or above, NumPy 1.11 or above, and Scipy.
Note that `MinHash LSH`_ and `MinHash LSH Ensemble`_ also support Redis and Cassandra
storage layer (see `MinHash LSH at Scale`_).
Install
-------
To install datasketch using ``pip``:
::
pip install datasketch
This will also install NumPy as dependency.
To install with Redis dependency:
::
pip install datasketch[redis]
To install with Cassandra dependency:
::
pip install datasketch[cassandra]
.. _`MinHash`: https://ekzhu.github.io/datasketch/minhash.html
.. _`Weighted MinHash`: https://ekzhu.github.io/datasketch/weightedminhash.html
.. _`HyperLogLog`: https://ekzhu.github.io/datasketch/hyperloglog.html
.. _`HyperLogLog++`: https://ekzhu.github.io/datasketch/hyperloglog.html#hyperloglog-plusplus
.. _`MinHash LSH`: https://ekzhu.github.io/datasketch/lsh.html
.. _`MinHash LSH Forest`: https://ekzhu.github.io/datasketch/lshforest.html
.. _`MinHash LSH Ensemble`: https://ekzhu.github.io/datasketch/lshensemble.html
.. _`Minhash LSH at Scale`: http://ekzhu.github.io/datasketch/lsh.html#minhash-lsh-at-scale
.. _`HNSW`: https://ekzhu.github.io/datasketch/documentation.html#hnsw
Raw data
{
"_id": null,
"home_page": "https://ekzhu.github.io/datasketch",
"name": "datasketch",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "database datamining",
"author": "ekzhu",
"author_email": "ekzhu@cs.toronto.edu",
"download_url": "https://files.pythonhosted.org/packages/21/f4/129a60f96c2739d11153c97b93f009177024dcb127f3471375eae1745fae/datasketch-1.6.3.tar.gz",
"platform": null,
"description": "datasketch: Big Data Looks Small\n================================\n\n.. image:: https://github.com/ekzhu/datasketch/workflows/Python%20package/badge.svg\n :target: https://github.com/ekzhu/datasketch/actions\n\n.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.290602.svg\n :target: https://doi.org/10.5281/zenodo.290602\n\ndatasketch gives you probabilistic data structures that can process and\nsearch very large amount of data super fast, with little loss of\naccuracy.\n\nThis package contains the following data sketches:\n\n+-------------------------+-----------------------------------------------+\n| Data Sketch | Usage |\n+=========================+===============================================+\n| `MinHash`_ | estimate Jaccard similarity and cardinality |\n+-------------------------+-----------------------------------------------+\n| `Weighted MinHash`_ | estimate weighted Jaccard similarity |\n+-------------------------+-----------------------------------------------+\n| `HyperLogLog`_ | estimate cardinality |\n+-------------------------+-----------------------------------------------+\n| `HyperLogLog++`_ | estimate cardinality |\n+-------------------------+-----------------------------------------------+\n\nThe following indexes for data sketches are provided to support\nsub-linear query time:\n\n+---------------------------+-----------------------------+------------------------+\n| Index | For Data Sketch | Supported Query Type |\n+===========================+=============================+========================+\n| `MinHash LSH`_ | MinHash, Weighted MinHash | Jaccard Threshold |\n+---------------------------+-----------------------------+------------------------+\n| `MinHash LSH Forest`_ | MinHash, Weighted MinHash | Jaccard Top-K |\n+---------------------------+-----------------------------+------------------------+\n| `MinHash LSH Ensemble`_ | MinHash | Containment Threshold |\n+---------------------------+-----------------------------+------------------------+\n| `HNSW`_ | Any | Custom Metric Top-K |\n+---------------------------+-----------------------------+------------------------+\n\ndatasketch must be used with Python 3.7 or above, NumPy 1.11 or above, and Scipy. \n\nNote that `MinHash LSH`_ and `MinHash LSH Ensemble`_ also support Redis and Cassandra \nstorage layer (see `MinHash LSH at Scale`_).\n\nInstall\n-------\n\nTo install datasketch using ``pip``:\n\n::\n\n pip install datasketch\n\nThis will also install NumPy as dependency.\n\nTo install with Redis dependency:\n\n::\n\n pip install datasketch[redis]\n\nTo install with Cassandra dependency:\n\n::\n\n pip install datasketch[cassandra]\n\n\n.. _`MinHash`: https://ekzhu.github.io/datasketch/minhash.html\n.. _`Weighted MinHash`: https://ekzhu.github.io/datasketch/weightedminhash.html\n.. _`HyperLogLog`: https://ekzhu.github.io/datasketch/hyperloglog.html\n.. _`HyperLogLog++`: https://ekzhu.github.io/datasketch/hyperloglog.html#hyperloglog-plusplus\n.. _`MinHash LSH`: https://ekzhu.github.io/datasketch/lsh.html\n.. _`MinHash LSH Forest`: https://ekzhu.github.io/datasketch/lshforest.html\n.. _`MinHash LSH Ensemble`: https://ekzhu.github.io/datasketch/lshensemble.html\n.. _`Minhash LSH at Scale`: http://ekzhu.github.io/datasketch/lsh.html#minhash-lsh-at-scale\n.. _`HNSW`: https://ekzhu.github.io/datasketch/documentation.html#hnsw\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Probabilistic data structures for processing and searching very large datasets",
"version": "1.6.3",
"project_urls": {
"Homepage": "https://ekzhu.github.io/datasketch",
"Source": "https://github.com/ekzhu/datasketch"
},
"split_keywords": [
"database",
"datamining"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a1960f8b80902fe8ade9b513923f7a0231439c96f10f4baf8c886a4697c2e4f7",
"md5": "3fac9249440a087fb5aad40a3e86f2e4",
"sha256": "905aa55596072a5a8f0f6c8cfa157bf1a3e18cce4f1744432c3c83623f0dfee4"
},
"downloads": -1,
"filename": "datasketch-1.6.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3fac9249440a087fb5aad40a3e86f2e4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 88387,
"upload_time": "2023-09-12T08:11:36",
"upload_time_iso_8601": "2023-09-12T08:11:36.626013Z",
"url": "https://files.pythonhosted.org/packages/a1/96/0f8b80902fe8ade9b513923f7a0231439c96f10f4baf8c886a4697c2e4f7/datasketch-1.6.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "21f4129a60f96c2739d11153c97b93f009177024dcb127f3471375eae1745fae",
"md5": "b8398cd5b397ddc67a32019a9140f31d",
"sha256": "bf7a8ab415140fa1e27c53797435f14366331713c20c4d26c28c1e4dc7189417"
},
"downloads": -1,
"filename": "datasketch-1.6.3.tar.gz",
"has_sig": false,
"md5_digest": "b8398cd5b397ddc67a32019a9140f31d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 91392,
"upload_time": "2023-09-12T08:11:38",
"upload_time_iso_8601": "2023-09-12T08:11:38.483668Z",
"url": "https://files.pythonhosted.org/packages/21/f4/129a60f96c2739d11153c97b93f009177024dcb127f3471375eae1745fae/datasketch-1.6.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-12 08:11:38",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ekzhu",
"github_project": "datasketch",
"travis_ci": true,
"coveralls": false,
"github_actions": true,
"lcname": "datasketch"
}
JSON
