# Amazon DenseClus
<p align="left">
<a href="https://github.com/awslabs/amazon-denseclus/actions/workflows/tests.yml"><img alt="build" src="https://github.com/awslabs/amazon-denseclus/actions/workflows/cd.yml/badge.svg"></a>
<a><img alt="total download" src="https://static.pepy.tech/personalized-badge/amazon-denseclus?period=total&units=international_system&left_color=black&right_color=green&left_text=Total Downloads"></a>
<a><img alt="month download" src="https://static.pepy.tech/personalized-badge/amazon-denseclus?period=month&units=international_system&left_color=black&right_color=green&left_text=Monthly Downloads"></a>
<a><img alt="weekly download" src="https://static.pepy.tech/personalized-badge/amazon-denseclus?period=week&units=international_system&left_color=black&right_color=green&left_text=Weekly Downloads"></a>
<a href="https://badge.fury.io/py/Amazon-DenseClus"><img alt="PyPI version" src="https://badge.fury.io/py/Amazon-DenseClus.svg"></a>
<a><img alt="PyPI - Python Version" src="https://img.shields.io/pypi/pyversions/Amazon-DenseClus"></a>
<a><img alt="PyPI - Wheel" src="https://img.shields.io/pypi/wheel/Amazon-DenseClus"></a>
<a><img alt="PyPI - License" src="https://img.shields.io/pypi/l/Amazon-DenseClus"></a>
<a href="https://github.com/psf/black"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg"></a>
<a href="https://github.com/marketplace/actions/super-linter"><img alt="Github Super-Linter" src="https://github.com/awslabs/amazon-denseclus/workflows/Lint%20Code%20Base/badge.svg"></a>
</p>
DenseClus is a Python module for clustering mixed type data using [UMAP](https://github.com/lmcinnes/umap) and [HDBSCAN](https://github.com/scikit-learn-contrib/hdbscan). Allowing for both categorical and numerical data, DenseClus makes it possible to incorporate all features in clustering.
## Installation
```bash
python3 -m pip install amazon-denseclus
```
## Quick Start
DenseClus requires a Panda's dataframe as input with both numerical and categorical columns.
All preprocessing and extraction are done under the hood, just call fit and then retrieve the clusters!
```python
from denseclus import DenseClus
from denseclus.utils import make_dataframe
df = make_dataframe()
clf = DenseClus(df)
clf.fit(df)
scores = clf.evaluate()
print(scores[0:10])
```
## Usage
### Prediction
DenseClus uses a `predict` method when `umap_combine_method` is set to `ensemble`.
Results are return in 2d array with the first part being the labels and the second part the probabilities.
```python
from denseclus import DenseClus
from denseclus.utils import make_dataframe
RANDOM_STATE = 10
df = make_dataframe(random_state=RANDOM_STATE)
train = df.sample(frac=0.8, random_state=RANDOM_STATE)
test = df.drop(train.index)
clf = DenseClus(random_state=RANDOM_STATE, umap_combine_method='ensemble')
clf.fit(train)
predictions = clf.predict(test)
print(predictions) # labels, probabilities
```
### On Combination Method
For a slower but more **stable** results select `intersection_union_mapper` to combine embedding layers via a third UMAP, which will provide equal weight to both numerics and categoriel columns. By default, you are setting the random seed which eliminates the ability for UMAP to run in parallel but will help circumevent some of [the randomness](https://umap-learn.readthedocs.io/en/latest/reproducibility.html) of the algorithm.
```python
clf = DenseClus(
umap_combine_method="intersection_union_mapper",
)
```
### To Use with GPU with Ensemble
To use with gpu first have [rapids installed](https://docs.rapids.ai/install#selector).
You can do this as setup by providing cuda verision.
`pip install denseclus[gpu-cu12]`
Then to run:
```python
clf = DenseClus(
umap_combine_method="ensemble",
use_gpu=True
)
```
### Advanced Usage
For advanced users, it's possible to select more fine-grained control of the underlying algorithms by passing
dictionaries into `DenseClus` class for either UMAP or HDBSCAN.
For example:
```python
from denseclus import DenseClus
from denseclus.utils import make_dataframe
umap_params = {
"categorical": {"n_neighbors": 15, "min_dist": 0.1},
"numerical": {"n_neighbors": 20, "min_dist": 0.1},
}
hdbscan_params = {"min_cluster_size": 10}
df = make_dataframe()
clf = DenseClus(umap_combine_method="union"
, umap_params=umap_params
, hdbscan_params=hdbscan_params
, random_state=None) # this will run in parallel
clf.fit(df)
```
## Examples
### Notebooks
A hands-on example with an overview of how to use is currently available in the form of a [Example Jupyter Notebook](/notebooks/01_DenseClusExampleNB.ipynb).
Should you need to tune HDBSCAN, here is an optional approach: [Tuning with HDBSCAN Notebook](/notebooks/02_TuningwithHDBSCAN.ipynb)
Should you need to validate UMAP emeddings, there is an approach to do so in the [Validation for UMAP Notebook](/notebooks/03_ValidationForUMAP.ipynb)
### Blogs
[AWS Blog: Introducing DenseClus, an open source clustering package for mixed-type data](https://aws.amazon.com/blogs/opensource/introducing-denseclus-an-open-source-clustering-package-for-mixed-type-data/)
[TDS Blog: How To Tune HDBSCAN](https://towardsdatascience.com/tuning-with-hdbscan-149865ac2970)
[TDS Blog: On the Validation of UMAP](https://towardsdatascience.com/on-the-validating-umap-embeddings-2c8907588175)
## References
```bibtex
@article{mcinnes2018umap-software,
title={UMAP: Uniform Manifold Approximation and Projection},
author={McInnes, Leland and Healy, John and Saul, Nathaniel and Grossberger, Lukas},
journal={The Journal of Open Source Software},
volume={3},
number={29},
pages={861},
year={2018}
}
```
```bibtex
@article{mcinnes2017hdbscan,
title={hdbscan: Hierarchical density based clustering},
author={McInnes, Leland and Healy, John and Astels, Steve},
journal={The Journal of Open Source Software},
volume={2},
number={11},
pages={205},
year={2017}
}
```
Raw data
{
"_id": null,
"home_page": "https://github.com/awslabs/amazon-denseclus",
"name": "amazon-denseclus",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10, <3.12",
"maintainer_email": "",
"keywords": "amazon dense clustering",
"author": "Charles Frenzel & Baichuan Sun",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/70/45/ba847b60fea69e7f416efd56841d72502549ea3421ca1c665520ee18f576/amazon_denseclus-0.2.2.tar.gz",
"platform": "any",
"description": "\n# Amazon DenseClus\n\n<p align=\"left\">\n<a href=\"https://github.com/awslabs/amazon-denseclus/actions/workflows/tests.yml\"><img alt=\"build\" src=\"https://github.com/awslabs/amazon-denseclus/actions/workflows/cd.yml/badge.svg\"></a>\n<a><img alt=\"total download\" src=\"https://static.pepy.tech/personalized-badge/amazon-denseclus?period=total&units=international_system&left_color=black&right_color=green&left_text=Total Downloads\"></a>\n<a><img alt=\"month download\" src=\"https://static.pepy.tech/personalized-badge/amazon-denseclus?period=month&units=international_system&left_color=black&right_color=green&left_text=Monthly Downloads\"></a>\n<a><img alt=\"weekly download\" src=\"https://static.pepy.tech/personalized-badge/amazon-denseclus?period=week&units=international_system&left_color=black&right_color=green&left_text=Weekly Downloads\"></a>\n<a href=\"https://badge.fury.io/py/Amazon-DenseClus\"><img alt=\"PyPI version\" src=\"https://badge.fury.io/py/Amazon-DenseClus.svg\"></a>\n<a><img alt=\"PyPI - Python Version\" src=\"https://img.shields.io/pypi/pyversions/Amazon-DenseClus\"></a>\n<a><img alt=\"PyPI - Wheel\" src=\"https://img.shields.io/pypi/wheel/Amazon-DenseClus\"></a>\n<a><img alt=\"PyPI - License\" src=\"https://img.shields.io/pypi/l/Amazon-DenseClus\"></a>\n<a href=\"https://github.com/psf/black\"><img alt=\"Code style: black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg\"></a>\n<a href=\"https://github.com/marketplace/actions/super-linter\"><img alt=\"Github Super-Linter\" src=\"https://github.com/awslabs/amazon-denseclus/workflows/Lint%20Code%20Base/badge.svg\"></a>\n</p>\n\n\n\nDenseClus is a Python module for clustering mixed type data using [UMAP](https://github.com/lmcinnes/umap) and [HDBSCAN](https://github.com/scikit-learn-contrib/hdbscan). Allowing for both categorical and numerical data, DenseClus makes it possible to incorporate all features in clustering.\n\n## Installation\n\n```bash\npython3 -m pip install amazon-denseclus\n```\n\n## Quick Start\n\nDenseClus requires a Panda's dataframe as input with both numerical and categorical columns.\nAll preprocessing and extraction are done under the hood, just call fit and then retrieve the clusters!\n\n```python\nfrom denseclus import DenseClus\nfrom denseclus.utils import make_dataframe\n\n\ndf = make_dataframe()\nclf = DenseClus(df)\nclf.fit(df)\n\nscores = clf.evaluate()\nprint(scores[0:10])\n```\n\n\n## Usage\n\n### Prediction\n\nDenseClus uses a `predict` method when `umap_combine_method` is set to `ensemble`.\nResults are return in 2d array with the first part being the labels and the second part the probabilities.\n\n```python\nfrom denseclus import DenseClus\nfrom denseclus.utils import make_dataframe\n\nRANDOM_STATE = 10\n\ndf = make_dataframe(random_state=RANDOM_STATE)\ntrain = df.sample(frac=0.8, random_state=RANDOM_STATE)\ntest = df.drop(train.index)\nclf = DenseClus(random_state=RANDOM_STATE, umap_combine_method='ensemble')\nclf.fit(train)\n\npredictions = clf.predict(test)\nprint(predictions) # labels, probabilities\n```\n\n\n### On Combination Method\n\nFor a slower but more **stable** results select `intersection_union_mapper` to combine embedding layers via a third UMAP, which will provide equal weight to both numerics and categoriel columns. By default, you are setting the random seed which eliminates the ability for UMAP to run in parallel but will help circumevent some of [the randomness](https://umap-learn.readthedocs.io/en/latest/reproducibility.html) of the algorithm.\n\n```python\nclf = DenseClus(\n umap_combine_method=\"intersection_union_mapper\",\n)\n```\n\n### To Use with GPU with Ensemble\n\nTo use with gpu first have [rapids installed](https://docs.rapids.ai/install#selector).\nYou can do this as setup by providing cuda verision.\n`pip install denseclus[gpu-cu12]`\n\nThen to run:\n\n```python\nclf = DenseClus(\n umap_combine_method=\"ensemble\",\n use_gpu=True\n)\n```\n\n\n### Advanced Usage\n\nFor advanced users, it's possible to select more fine-grained control of the underlying algorithms by passing\ndictionaries into `DenseClus` class for either UMAP or HDBSCAN.\n\nFor example:\n```python\nfrom denseclus import DenseClus\nfrom denseclus.utils import make_dataframe\n\numap_params = {\n \"categorical\": {\"n_neighbors\": 15, \"min_dist\": 0.1},\n \"numerical\": {\"n_neighbors\": 20, \"min_dist\": 0.1},\n}\nhdbscan_params = {\"min_cluster_size\": 10}\n\ndf = make_dataframe()\n\nclf = DenseClus(umap_combine_method=\"union\"\n , umap_params=umap_params\n , hdbscan_params=hdbscan_params\n , random_state=None) # this will run in parallel\n\nclf.fit(df)\n```\n\n\n## Examples\n\n### Notebooks\n\nA hands-on example with an overview of how to use is currently available in the form of a [Example Jupyter Notebook](/notebooks/01_DenseClusExampleNB.ipynb).\n\nShould you need to tune HDBSCAN, here is an optional approach: [Tuning with HDBSCAN Notebook](/notebooks/02_TuningwithHDBSCAN.ipynb)\n\nShould you need to validate UMAP emeddings, there is an approach to do so in the [Validation for UMAP Notebook](/notebooks/03_ValidationForUMAP.ipynb)\n\n### Blogs\n\n\n[AWS Blog: Introducing DenseClus, an open source clustering package for mixed-type data](https://aws.amazon.com/blogs/opensource/introducing-denseclus-an-open-source-clustering-package-for-mixed-type-data/)\n\n[TDS Blog: How To Tune HDBSCAN](https://towardsdatascience.com/tuning-with-hdbscan-149865ac2970)\n\n[TDS Blog: On the Validation of UMAP](https://towardsdatascience.com/on-the-validating-umap-embeddings-2c8907588175)\n\n\n\n## References\n\n```bibtex\n@article{mcinnes2018umap-software,\n title={UMAP: Uniform Manifold Approximation and Projection},\n author={McInnes, Leland and Healy, John and Saul, Nathaniel and Grossberger, Lukas},\n journal={The Journal of Open Source Software},\n volume={3},\n number={29},\n pages={861},\n year={2018}\n}\n```\n\n```bibtex\n@article{mcinnes2017hdbscan,\n title={hdbscan: Hierarchical density based clustering},\n author={McInnes, Leland and Healy, John and Astels, Steve},\n journal={The Journal of Open Source Software},\n volume={2},\n number={11},\n pages={205},\n year={2017}\n}\n```\n",
"bugtrack_url": null,
"license": "",
"summary": "Dense Clustering for Mixed Data Types",
"version": "0.2.2",
"project_urls": {
"Bug Tracker": "https://github.com/awslabs/amazon-denseclus/issues",
"Documentation": "https://github.com/awslabs/amazon-denseclus/notebooks",
"Homepage": "https://github.com/awslabs/amazon-denseclus",
"Source Code": "https://github.com/awslabs/amazon-denseclus"
},
"split_keywords": [
"amazon",
"dense",
"clustering"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f186d6104f070ca50bcb55c6a939ab9bd3fdd0d317db9d93e09ea7be8450e7d2",
"md5": "55b5969058c0588e564427459857a1eb",
"sha256": "0b7a4bd6dca2e141ad0302bffd8f517ae73299dabc25082dc903acbf63190efb"
},
"downloads": -1,
"filename": "amazon_denseclus-0.2.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "55b5969058c0588e564427459857a1eb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10, <3.12",
"size": 13489,
"upload_time": "2024-02-29T20:30:52",
"upload_time_iso_8601": "2024-02-29T20:30:52.491706Z",
"url": "https://files.pythonhosted.org/packages/f1/86/d6104f070ca50bcb55c6a939ab9bd3fdd0d317db9d93e09ea7be8450e7d2/amazon_denseclus-0.2.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7045ba847b60fea69e7f416efd56841d72502549ea3421ca1c665520ee18f576",
"md5": "c62e664d1f88e49775f8a397528e10e0",
"sha256": "e6eaf3e8782e2b1061a8cc57419a3dc427dce58c26f6bac5082911bce5da2309"
},
"downloads": -1,
"filename": "amazon_denseclus-0.2.2.tar.gz",
"has_sig": false,
"md5_digest": "c62e664d1f88e49775f8a397528e10e0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10, <3.12",
"size": 2105863,
"upload_time": "2024-02-29T20:30:54",
"upload_time_iso_8601": "2024-02-29T20:30:54.753094Z",
"url": "https://files.pythonhosted.org/packages/70/45/ba847b60fea69e7f416efd56841d72502549ea3421ca1c665520ee18f576/amazon_denseclus-0.2.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-29 20:30:54",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "awslabs",
"github_project": "amazon-denseclus",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"tox": true,
"lcname": "amazon-denseclus"
}
JSON
