PhenoGraph for Python3
======================
[PhenoGraph](http://www.cell.com/cell/abstract/S0092-8674(15)00637-6) is a clustering method designed for high-dimensional single-cell data. It works by creating a graph ("network") representing phenotypic similarities between cells and then identifying communities in this graph.
This software package includes compiled binaries that run community detection based on C++ code written by E. Lefebvre and J.-L. Guillaume in 2008 (["Louvain method"](https://sites.google.com/site/findcommunities/)). The code has been altered to interface more efficiently with the Python code here. It should work on reasonably current Linux, Mac and Windows machines.
To install PhenoGraph, simply run the setup script:
pip install PhenoGraph
Expected use is within a script or interactive kernel running Python `3.x`. Data are expected to be passed as a `numpy.ndarray`. When applicable, the code uses CPU multicore parallelism via `multiprocessing`.
To run basic clustering:
import phenograph
communities, graph, Q = phenograph.cluster(data)
For a dataset of *N* rows, `communities` will be a length *N* vector of integers specifying a community assignment for each row in the data. Any rows assigned `-1` were identified as *outliers* and should not be considered as a member of any community. `graph` is a *N* x *N* `scipy.sparse` matrix representing the weighted graph used for community detection.
`Q` is the modularity score for `communities` as applied to `graph`.
If you use PhenoGraph in work you publish, please cite our publication:
@article{Levine_PhenoGraph_2015,
doi = {10.1016/j.cell.2015.05.047},
url = {http://dx.doi.org/10.1016/j.cell.2015.05.047},
year = {2015},
month = {jul},
publisher = {Elsevier {BV}},
volume = {162},
number = {1},
pages = {184--197},
author = {Jacob H. Levine and Erin F. Simonds and Sean C. Bendall and Kara L. Davis and El-ad D. Amir and Michelle D. Tadmor and Oren Litvin and Harris G. Fienberg and Astraea Jager and Eli R. Zunder and Rachel Finck and Amanda L. Gedman and Ina Radtke and James R. Downing and Dana Pe'er and Garry P. Nolan},
title = {Data-Driven Phenotypic Dissection of {AML} Reveals Progenitor-like Cells that Correlate with Prognosis},
journal = {Cell}
}
Release Notes
-------------
### Version 1.5.7
* Updated leidenalg and scipy version requirements, revised parallel jaccard to support scipy==1.5.1, and created a test collection for use with pytest (see [below](#running-the-unit-tests)).
* Added [PhenoGraph clustering tutorial](examples/tutorial_pbmc3k.ipynb) with PBMC3K dataset from 10X Genomics (dataset included).
### Version 1.5.6
* Fix the multiprocessing code that doesn't close/join the pool.
### Version 1.5.5
* Exposed parameter `n_iterations` for Leiden, along with minor fixes to manage sorting parallelism, and updated documentation of the clustering and sorting methods.
### Version 1.5.4
* Faster and more efficient sorting by size of clusters, for large nearest neighbours graph, implementing multiprocessing and faster methods for sorting.
### Version 1.5.3
* Phenograph supports now [**Leiden**](https://www.nature.com/articles/s41598-019-41695-z) algorithm for community detection. The new feature can be called from `phenograph.cluster`, by choosing `leiden` as the clustering algorithm.
### Version 1.5.2
* Include simple parallel implementation of brute force nearest neighbors search using scipy's `cdist` and `multiprocessing`. This may be more efficient than `kdtree` on very large high-dimensional data sets
and avoids memory issues that arise in `sklearn`'s implementation.
* Refactor `parallel_jaccard_kernel` to remove unnecessary use of `ctypes` and `multiprocessing.Array`.
### Version 1.5.1
* Make `louvain_time_limit` a parameter to `phenograph.cluster`.
### Version 1.5
* `phenograph.cluster` can now take as input a square sparse matrix, which will be interpreted as a k-nearest neighbor graph.
Note that this graph _must_ have uniform degree (i.e. the same value of k at every point).
* The default `time_limit` for Louvain iterations has been increased to a more generous 2000 seconds (~half hour).
### Version 1.4.1
* After observing inconsistent behavior of sklearn.NearestNeighbors with respect to inclusion of self-neighbors,
the code now checks that self-neighbors have been included before deleting those entries.
### Version 1.4
* The dependence on IPython and/or ipyparallel has been removed. Instead the native `multiprocessing` package is used.
* Multiple CPUs are used by default for computation of nearest neighbors and Jaccard graph.
### Version 1.3
* Proper support for Linux.
---
Running the Unit Tests
---------------
Unit tests for assessing the functionality of each module are included in the 'tests\' directory. In addition to the dependencies required by PhenoGraph, to run these tests, you must first install the [pytest](https://docs.pytest.org) module.
If your system uses Python >= 3.8.0 or greater, install pytest with:
pip install pytest #Python >= 3.8.0
Otherwise, install pytest with:
pip install pytest==6.0.2 #Python < 3.8.0
Once pytest is installed, navigate to the 'PhenoGraph/' directory and run with:
pytest
All tests should pass with no warnings.
Troubleshooting
---------------
### Notebook freezes after several attempts of running PhenoGraph using Jypyter Notebook
* Running `PhenoGraph` from a Jupyter Notebook repeatedly on macOS Catalina, but not Mojave, using Python 3.7.6, causes a hang and the notebook becomes unresponsive, even for a basic matrix of nearest neighbors. However, this issue was not reproducible in command line using `Python` interpreter in both Catalina and Mojave platforms, without using Jupyter Notebook.
It was found that attempting to plot principal components using
```
:func:`~matplotlib.pyplot.scatter`
```
in Jupyter Notebook causes a freeze, and `PhenoGraph` becomes unresponsive unless the kernel is restarted. When removing this line of code, everything goes back to normal and the Jupyter Notebook stopes crashing with repeated runs of `PhenoGraph`.
### Architecture related error
* When attempting to process very large nearest neighbours graph, _e.g._ a 2000000 `x` 2000000 kNN graph matrix with 300 nearest neighbours, a `struct.error()` is raised:
```python
struct.error: 'i' format requires -2147483648 <= number <= 2147483647
```
This issue was reported on [stackoverflow](https://stackoverflow.com/questions/47776486/python-struct-error-i-format-requires-2147483648-number-2147483647) and it's related to the multiprocessing while building the Jaccard object.
The `struct.error()` has been fixed in python >= 3.8.0.
### `leidenalg` inside conda environment
* When using `PhenoGraph` inside a conda environment `leiden` takes longer to complete for larger samples compared to the system Python.
### `numpy.ufunc` runtime warning when running pytest
* When running unit tests, pytest may deliver the following warning `RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility`. This is caused by an incompatability between newer versions of the pytest module and an older version of Python. Check your Python version with:
```
python --version
```
If using Python >= 3.8.0, all version of pytest are compatible. If using Python < 3.8.0, downgrade to pytest version 6.0.2 (see above).
Raw data
{
"_id": null,
"home_page": "https://github.com/dpeerlab/PhenoGraph.git",
"name": "PhenoGraph",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "",
"author": "Jacob Levine",
"author_email": "jl3545@columbia.edu",
"download_url": "https://files.pythonhosted.org/packages/75/97/c1077df94cb0e1ebd201d67e52c438810fdaa393dc4c663b3bc651e497b8/PhenoGraph-1.5.7.tar.gz",
"platform": "",
"description": "PhenoGraph for Python3\n======================\n\n[PhenoGraph](http://www.cell.com/cell/abstract/S0092-8674(15)00637-6) is a clustering method designed for high-dimensional single-cell data. It works by creating a graph (\"network\") representing phenotypic similarities between cells and then identifying communities in this graph.\n\nThis software package includes compiled binaries that run community detection based on C++ code written by E. Lefebvre and J.-L. Guillaume in 2008 ([\"Louvain method\"](https://sites.google.com/site/findcommunities/)). The code has been altered to interface more efficiently with the Python code here. It should work on reasonably current Linux, Mac and Windows machines.\n\nTo install PhenoGraph, simply run the setup script:\n\n pip install PhenoGraph\n\n\nExpected use is within a script or interactive kernel running Python `3.x`. Data are expected to be passed as a `numpy.ndarray`. When applicable, the code uses CPU multicore parallelism via `multiprocessing`.\n\nTo run basic clustering:\n\n import phenograph\n communities, graph, Q = phenograph.cluster(data)\n\nFor a dataset of *N* rows, `communities` will be a length *N* vector of integers specifying a community assignment for each row in the data. Any rows assigned `-1` were identified as *outliers* and should not be considered as a member of any community. `graph` is a *N* x *N* `scipy.sparse` matrix representing the weighted graph used for community detection.\n`Q` is the modularity score for `communities` as applied to `graph`.\n\nIf you use PhenoGraph in work you publish, please cite our publication:\n\n @article{Levine_PhenoGraph_2015,\n doi = {10.1016/j.cell.2015.05.047},\n url = {http://dx.doi.org/10.1016/j.cell.2015.05.047},\n year = {2015},\n month = {jul},\n publisher = {Elsevier {BV}},\n volume = {162},\n number = {1},\n pages = {184--197},\n author = {Jacob H. Levine and Erin F. Simonds and Sean C. Bendall and Kara L. Davis and El-ad D. Amir and Michelle D. Tadmor and Oren Litvin and Harris G. Fienberg and Astraea Jager and Eli R. Zunder and Rachel Finck and Amanda L. Gedman and Ina Radtke and James R. Downing and Dana Pe'er and Garry P. Nolan},\n title = {Data-Driven Phenotypic Dissection of {AML} Reveals Progenitor-like Cells that Correlate with Prognosis},\n journal = {Cell}\n }\n\nRelease Notes\n-------------\n\n### Version 1.5.7\n\n* Updated leidenalg and scipy version requirements, revised parallel jaccard to support scipy==1.5.1, and created a test collection for use with pytest (see [below](#running-the-unit-tests)).\n* Added [PhenoGraph clustering tutorial](examples/tutorial_pbmc3k.ipynb) with PBMC3K dataset from 10X Genomics (dataset included).\n\n### Version 1.5.6\n\n* Fix the multiprocessing code that doesn't close/join the pool.\n\n### Version 1.5.5\n\n* Exposed parameter `n_iterations` for Leiden, along with minor fixes to manage sorting parallelism, and updated documentation of the clustering and sorting methods.\n\n### Version 1.5.4\n\n* Faster and more efficient sorting by size of clusters, for large nearest neighbours graph, implementing multiprocessing and faster methods for sorting.\n\n### Version 1.5.3\n\n* Phenograph supports now [**Leiden**](https://www.nature.com/articles/s41598-019-41695-z) algorithm for community detection. The new feature can be called from `phenograph.cluster`, by choosing `leiden` as the clustering algorithm.\n\n### Version 1.5.2\n\n* Include simple parallel implementation of brute force nearest neighbors search using scipy's `cdist` and `multiprocessing`. This may be more efficient than `kdtree` on very large high-dimensional data sets\n and avoids memory issues that arise in `sklearn`'s implementation.\n * Refactor `parallel_jaccard_kernel` to remove unnecessary use of `ctypes` and `multiprocessing.Array`.\n\n### Version 1.5.1\n\n* Make `louvain_time_limit` a parameter to `phenograph.cluster`.\n\n### Version 1.5\n\n * `phenograph.cluster` can now take as input a square sparse matrix, which will be interpreted as a k-nearest neighbor graph.\n Note that this graph _must_ have uniform degree (i.e. the same value of k at every point).\n * The default `time_limit` for Louvain iterations has been increased to a more generous 2000 seconds (~half hour).\n\n### Version 1.4.1\n\n* After observing inconsistent behavior of sklearn.NearestNeighbors with respect to inclusion of self-neighbors,\n the code now checks that self-neighbors have been included before deleting those entries.\n\n### Version 1.4\n\n* The dependence on IPython and/or ipyparallel has been removed. Instead the native `multiprocessing` package is used.\n* Multiple CPUs are used by default for computation of nearest neighbors and Jaccard graph.\n\n### Version 1.3\n\n* Proper support for Linux.\n\n---\nRunning the Unit Tests\n---------------\n\nUnit tests for assessing the functionality of each module are included in the 'tests\\' directory. In addition to the dependencies required by PhenoGraph, to run these tests, you must first install the [pytest](https://docs.pytest.org) module.\n\nIf your system uses Python >= 3.8.0 or greater, install pytest with:\n\n pip install pytest #Python >= 3.8.0\n\nOtherwise, install pytest with:\n\n pip install pytest==6.0.2 #Python < 3.8.0\n\nOnce pytest is installed, navigate to the 'PhenoGraph/' directory and run with:\n\n pytest\n\nAll tests should pass with no warnings.\n\nTroubleshooting\n---------------\n\n### Notebook freezes after several attempts of running PhenoGraph using Jypyter Notebook\n\n* Running `PhenoGraph` from a Jupyter Notebook repeatedly on macOS Catalina, but not Mojave, using Python 3.7.6, causes a hang and the notebook becomes unresponsive, even for a basic matrix of nearest neighbors. However, this issue was not reproducible in command line using `Python` interpreter in both Catalina and Mojave platforms, without using Jupyter Notebook.\n\n It was found that attempting to plot principal components using\n\n ```\n :func:`~matplotlib.pyplot.scatter`\n ```\n\n in Jupyter Notebook causes a freeze, and `PhenoGraph` becomes unresponsive unless the kernel is restarted. When removing this line of code, everything goes back to normal and the Jupyter Notebook stopes crashing with repeated runs of `PhenoGraph`.\n\n### Architecture related error\n\n* When attempting to process very large nearest neighbours graph, _e.g._ a 2000000 `x` 2000000 kNN graph matrix with 300 nearest neighbours, a `struct.error()` is raised:\n\n ```python\n struct.error: 'i' format requires -2147483648 <= number <= 2147483647\n ```\n\n This issue was reported on [stackoverflow](https://stackoverflow.com/questions/47776486/python-struct-error-i-format-requires-2147483648-number-2147483647) and it's related to the multiprocessing while building the Jaccard object.\n\n The `struct.error()` has been fixed in python >= 3.8.0.\n\n### `leidenalg` inside conda environment\n\n* When using `PhenoGraph` inside a conda environment `leiden` takes longer to complete for larger samples compared to the system Python.\n\n### `numpy.ufunc` runtime warning when running pytest\n\n* When running unit tests, pytest may deliver the following warning `RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility`. This is caused by an incompatability between newer versions of the pytest module and an older version of Python. Check your Python version with:\n\n ```\n python --version\n ```\n\n If using Python >= 3.8.0, all version of pytest are compatible. If using Python < 3.8.0, downgrade to pytest version 6.0.2 (see above).\n\n\n\n",
"bugtrack_url": null,
"license": "LICENSE",
"summary": "Graph-based clustering for high-dimensional single-cell data",
"version": "1.5.7",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "4b87d668614c5ef3e2854d13dd7a32da",
"sha256": "62a0fb88e13e4409875da0d382f0783a63faa4913b79f61b99528eb79c93c2e2"
},
"downloads": -1,
"filename": "PhenoGraph-1.5.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4b87d668614c5ef3e2854d13dd7a32da",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 159587,
"upload_time": "2020-10-05T17:18:22",
"upload_time_iso_8601": "2020-10-05T17:18:22.108037Z",
"url": "https://files.pythonhosted.org/packages/fc/37/4aa1d8c2ded0c612031d32ad8606b6222243f9326ca28754122e306680be/PhenoGraph-1.5.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "1011f7305a7f014fe922f400579d45b5",
"sha256": "46b28f9e043a00deba53bb5f35dd84793669ab2bd4ce78900bf7f15f1321515a"
},
"downloads": -1,
"filename": "PhenoGraph-1.5.7.tar.gz",
"has_sig": false,
"md5_digest": "1011f7305a7f014fe922f400579d45b5",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 8449829,
"upload_time": "2020-10-05T17:18:24",
"upload_time_iso_8601": "2020-10-05T17:18:24.190777Z",
"url": "https://files.pythonhosted.org/packages/75/97/c1077df94cb0e1ebd201d67e52c438810fdaa393dc4c663b3bc651e497b8/PhenoGraph-1.5.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2020-10-05 17:18:24",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "dpeerlab",
"github_project": "PhenoGraph.git",
"lcname": "phenograph"
}
JSON
