# Overview
Cellxgene Gateway allows you to use the Cellxgene Server provided by the Chan Zuckerberg Institute (https://github.com/chanzuckerberg/cellxgene) with multiple datasets. It displays an index of available h5ad (anndata) files. When a user clicks on a file name, it launches a Cellxgene Server instance that loads that particular data file and once it is available proxies requests to that server.
[](https://codecov.io/gh/Novartis/cellxgene-gateway) [](https://pypi.org/project/cellxgene-gateway/) [](https://pypistats.org/packages/cellxgene-gateway)
# Running locally
## Prequisites
1. This project requires python 3.6 or higher. Please check your version with
```bash
$ python --version
```
2. It is also a good idea to set up a venv
```bash
python -m venv .cellxgene-gateway
source .cellxgene-gateway/bin/activate # type `deactivate` to deactivate the venv
```
## Install cellxgene-gateway
### Option 1: Pip Install from Github
```bash
pip install git+https://github.com/Novartis/cellxgene-gateway
```
Note: you may need to downgrade h5py with `pip install h5py==2.9.0` due to an [issue](https://github.com/theislab/scanpy/issues/832) in a dependency.
### Option 2: Install from PyPI
```bash
pip install cellxgene-gateway
```
## Running cellxgene gateway
1. Prepare a folder with .h5ad files, for example
```bash
mkdir ../cellxgene_data
wget https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad -O ../cellxgene_data/pbmc3k.h5ad
```
2. Set your environment variables correctly:
```bash
export CELLXGENE_DATA=../cellxgene_data # change this directory if you put data in a different place.
export CELLXGENE_LOCATION=`which cellxgene`
```
3. Now, execute the cellxgene gateway:
```bash
cellxgene-gateway
```
Here's what the environment variables mean:
* `CELLXGENE_LOCATION` - the location of the cellxgene executable, e.g. `~/anaconda2/envs/cellxgene/bin/cellxgene`
At least one of the following is required:
* `CELLXGENE_DATA` - a directory that can contain subdirectories with `.h5ad` data files, *without* trailing slash, e.g. `/mnt/cellxgene_data`
* `CELLXGENE_BUCKET` - an s3 bucket that can contain keys with `.h5ad` data files, e.g. `my-cellxgene-data-bucket`
Cellxgene Gateway is designed to make it easy to add additional data sources, please see the source code for gateway.py and the ItemSource interface in items/item_source.py
Optional environment variables:
* `CELLXGENE_ARGS` - catch-all variable that can be used to pass additional command line args to cellxgene server
* `EXTERNAL_HOST` - the hostname and port from the perspective of the web browser, typically `localhost:5005` if running locally. Defaults to "localhost:{GATEWAY_PORT}"
* `EXTERNAL_PROTOCOL` - typically http when running locally, can be https when deployed if the gateway is behind a load balancer or reverse proxy that performs https termination. Default value "http"
* `GATEWAY_IP` - ip addess of instance gateway is running on, mostly used to display SSH instructions. Defaults to `socket.gethostbyname(socket.gethostname())`
* `GATEWAY_PORT` - local port that the gateway should bind to, defaults to 5005
* `GATEWAY_EXPIRE_SECONDS` - time in seconds that a cellxgene process will remain idle before being terminated. Defaults to 3600 (one hour)
* `GATEWAY_EXTRA_SCRIPTS` - JSON array of script paths, will be embedded into each page and forwarded with `--scripts` to cellxgene server
* `GATEWAY_ENABLE_ANNOTATIONS` - Set to `true` or to `1` to enable cellxgene annotations and gene sets.
* `GATEWAY_ENABLE_BACKED_MODE` - Set to `true` or to `1` to load AnnData in file-backed mode. This saves memory and speeds up launch time but may reduce overall performance.
* `GATEWAY_LOG_LEVEL` - default is `INFO`. set to `DEBUG` to increase logging and to `WARNING` to decrease logging.
* `S3_ENABLE_LISTINGS_CACHE` - Set to `true` or to `1` to cache listings of S3 folders for performance. If the cache becomes stale, set `filecrawl.html?refresh=true` query parameter to refresh the cache.
If any of the following optional variables are set, [ProxyFix](https://werkzeug.palletsprojects.com/en/1.0.x/middleware/proxy_fix/) will be used.
* `PROXY_FIX_FOR` - Number of upstream proxies setting X-Forwarded-For
* `PROXY_FIX_PROTO` - Number of upstream proxies setting X-Forwarded-Proto
* `PROXY_FIX_HOST` - Number of upstream proxies setting X-Forwarded-Host
* `PROXY_FIX_PORT` - Number of upstream proxies setting X-Forwarded-Port
* `PROXY_FIX_PREFIX` - Number of upstream proxies setting X-Forwarded-Prefix
The defaults should be fine if you set up a venv and cellxgene_data folder as above.
## Running cellxgene-gateway with Docker
First, build Docker image:
```bash
docker build -t cellxgene-gateway .
```
Then, cellxgene-gateway can be launched as such:
```bash
docker run -it --rm \
-v <local_data_dir>:/cellxgene-data \
-p 5005:5005 \
cellxgene-gateway
```
Additional environment variables can be provided with the `-e` parameter:
```bash
docker run -it --rm \
-v ../cellxgene_data:/cellxgene-data \
-e GATEWAY_PORT=8080 \
-p 8080:8080 \
cellxgene-gateway
```
# Customization
The current paradigm for customization is to modify files during a build or deployment phase:
* To modify CSS or JS on particular gateway pages, overwrite or append to the templates
* To add script tags such as for user analytics to all pages, set GATEWAY_EXTRA_SCRIPTS
* these scripts will also be run on the pages served by cellxgene server via the --scripts parameter
* See https://github.com/chanzuckerberg/cellxgene/pull/680 for details on --scripts parameter
Currently we use a bash script that copies the gateway to a "build" directory before modifying templates with sed and the like. There is probably a better way.
# Development
We’re actively developing. Please see the "future work" section of the [wiki](https://github.com/Novartis/cellxgene-gateway/wiki#future-work). If you’re interested in being a contributor please reach out to [@alokito](https://github.com/alokito).
## Developer Install
If you want to develop the code, you will need to clone the repo. Make sure you have the prequesite listed above, then:
1. Clone the repo
```bash
git clone https://github.com/Novartis/cellxgene-gateway.git
cd cellxgene-gateway
```
2. Install requirements with
```bash
pip install -r requirements.txt
```
3. Install the gateway in developer mode
```bash
python setup.py develop
```
For convenience, the code repo includes a `run.sh.example` shell script to run the gateway.
4. Install pre-commit hooks
```bash
conda install -c conda-forge pre-commit
pre-commit install
```
## Running Tests
[](https://travis-ci.org/Novartis/cellxgene-gateway)
```bash
python -m unittest discover tests
```
## Code Coverage
```bash
coverage run -m unittest discover tests
coverage html
```
## Running Linters
pip install isort flake8 black
```bash
isort -rc . # rc means recursive, and was deprecated in dev version of isort
black .
```
# Getting Help
If you need help for any reason, please make a github ticket. One of the contributors should help you out.
# Releasing New Versions
## How to prepare for release
- Update Changelog.md and version number in __init__.py
- Cut a release on github
- Go to your project homepage on GitHub
- On right side, you will see [Releases](https://github.com/Novartis/cellxgene-gateway/releases) link. Click on it.
- Click on Draft a new release
- Fill in all the details
- Tag version should be the version number of your package release
- Release Title can be anything you want, but we use v0.3.11 (the same as the tag to be created on publish)
- Description should be changelog
- Click Publish release at the bottom of the page
- Now under Releases you can view all of your releases.
- Copy the download link (tar.gz) and save it somewhere
## How to publish to PyPI
Make sure your `.pypirc` is set up for testpypi and pypi index servers.
```bash
rm -rf dist
python setup.py sdist bdist_wheel
python -m twine upload --repository testpypi dist/*
python -m twine upload dist/*
```
# Contributors
* Niket Patel - https://github.com/NiketPatel9
* Alok Saldanha - https://github.com/alokito
* Yohann Potier - https://github.com/ypotier
Raw data
{
"_id": null,
"home_page": "http://github.com/Novartis/cellxgene-gateway",
"name": "cellxgene-gateway",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "visualization,genomics",
"author": "Niket Patel, Yohann Potier, Alok Saldanha",
"author_email": "alok.saldanha@novartis.com",
"download_url": "https://files.pythonhosted.org/packages/af/45/d065124ee1800d00db0ab126721f816082f22b873f37f315d40c29fc9e35/cellxgene-gateway-0.4.0.tar.gz",
"platform": null,
"description": "# Overview\n\nCellxgene Gateway allows you to use the Cellxgene Server provided by the Chan Zuckerberg Institute (https://github.com/chanzuckerberg/cellxgene) with multiple datasets. It displays an index of available h5ad (anndata) files. When a user clicks on a file name, it launches a Cellxgene Server instance that loads that particular data file and once it is available proxies requests to that server.\n\n[](https://codecov.io/gh/Novartis/cellxgene-gateway) [](https://pypi.org/project/cellxgene-gateway/) [](https://pypistats.org/packages/cellxgene-gateway)\n\n# Running locally\n\n## Prequisites\n\n1. This project requires python 3.6 or higher. Please check your version with\n\n```bash\n$ python --version\n```\n\n2. It is also a good idea to set up a venv\n\n```bash\npython -m venv .cellxgene-gateway\nsource .cellxgene-gateway/bin/activate # type `deactivate` to deactivate the venv\n```\n\n## Install cellxgene-gateway\n\n### Option 1: Pip Install from Github\n\n```bash\npip install git+https://github.com/Novartis/cellxgene-gateway\n```\nNote: you may need to downgrade h5py with `pip install h5py==2.9.0` due to an [issue](https://github.com/theislab/scanpy/issues/832) in a dependency.\n### Option 2: Install from PyPI\n\n```bash\npip install cellxgene-gateway\n```\n\n## Running cellxgene gateway\n\n1. Prepare a folder with .h5ad files, for example\n\n```bash\nmkdir ../cellxgene_data\nwget https://raw.githubusercontent.com/chanzuckerberg/cellxgene/master/example-dataset/pbmc3k.h5ad -O ../cellxgene_data/pbmc3k.h5ad\n```\n\n\n2. Set your environment variables correctly:\n\n```bash\nexport CELLXGENE_DATA=../cellxgene_data # change this directory if you put data in a different place.\nexport CELLXGENE_LOCATION=`which cellxgene`\n```\n\n3. Now, execute the cellxgene gateway:\n\n```bash\ncellxgene-gateway\n```\n\nHere's what the environment variables mean:\n\n* `CELLXGENE_LOCATION` - the location of the cellxgene executable, e.g. `~/anaconda2/envs/cellxgene/bin/cellxgene`\n\nAt least one of the following is required:\n* `CELLXGENE_DATA` - a directory that can contain subdirectories with `.h5ad` data files, *without* trailing slash, e.g. `/mnt/cellxgene_data`\n* `CELLXGENE_BUCKET` - an s3 bucket that can contain keys with `.h5ad` data files, e.g. `my-cellxgene-data-bucket`\nCellxgene Gateway is designed to make it easy to add additional data sources, please see the source code for gateway.py and the ItemSource interface in items/item_source.py\n\nOptional environment variables:\n* `CELLXGENE_ARGS` - catch-all variable that can be used to pass additional command line args to cellxgene server\n* `EXTERNAL_HOST` - the hostname and port from the perspective of the web browser, typically `localhost:5005` if running locally. Defaults to \"localhost:{GATEWAY_PORT}\"\n* `EXTERNAL_PROTOCOL` - typically http when running locally, can be https when deployed if the gateway is behind a load balancer or reverse proxy that performs https termination. Default value \"http\"\n* `GATEWAY_IP` - ip addess of instance gateway is running on, mostly used to display SSH instructions. Defaults to `socket.gethostbyname(socket.gethostname())`\n* `GATEWAY_PORT` - local port that the gateway should bind to, defaults to 5005\n* `GATEWAY_EXPIRE_SECONDS` - time in seconds that a cellxgene process will remain idle before being terminated. Defaults to 3600 (one hour)\n* `GATEWAY_EXTRA_SCRIPTS` - JSON array of script paths, will be embedded into each page and forwarded with `--scripts` to cellxgene server\n* `GATEWAY_ENABLE_ANNOTATIONS` - Set to `true` or to `1` to enable cellxgene annotations and gene sets.\n* `GATEWAY_ENABLE_BACKED_MODE` - Set to `true` or to `1` to load AnnData in file-backed mode. This saves memory and speeds up launch time but may reduce overall performance.\n* `GATEWAY_LOG_LEVEL` - default is `INFO`. set to `DEBUG` to increase logging and to `WARNING` to decrease logging.\n* `S3_ENABLE_LISTINGS_CACHE` - Set to `true` or to `1` to cache listings of S3 folders for performance. If the cache becomes stale, set `filecrawl.html?refresh=true` query parameter to refresh the cache.\n\nIf any of the following optional variables are set, [ProxyFix](https://werkzeug.palletsprojects.com/en/1.0.x/middleware/proxy_fix/) will be used.\n* `PROXY_FIX_FOR` - Number of upstream proxies setting X-Forwarded-For\n* `PROXY_FIX_PROTO` - Number of upstream proxies setting X-Forwarded-Proto\n* `PROXY_FIX_HOST` - Number of upstream proxies setting X-Forwarded-Host\n* `PROXY_FIX_PORT` - Number of upstream proxies setting X-Forwarded-Port\n* `PROXY_FIX_PREFIX` - Number of upstream proxies setting X-Forwarded-Prefix\n\nThe defaults should be fine if you set up a venv and cellxgene_data folder as above.\n\n## Running cellxgene-gateway with Docker\n\nFirst, build Docker image:\n\n```bash\ndocker build -t cellxgene-gateway .\n```\n\nThen, cellxgene-gateway can be launched as such:\n\n```bash\ndocker run -it --rm \\\n-v <local_data_dir>:/cellxgene-data \\\n-p 5005:5005 \\\ncellxgene-gateway\n```\n\nAdditional environment variables can be provided with the `-e` parameter:\n\n```bash\ndocker run -it --rm \\\n-v ../cellxgene_data:/cellxgene-data \\\n-e GATEWAY_PORT=8080 \\\n-p 8080:8080 \\\ncellxgene-gateway\n```\n\n# Customization\n\nThe current paradigm for customization is to modify files during a build or deployment phase:\n\n* To modify CSS or JS on particular gateway pages, overwrite or append to the templates\n* To add script tags such as for user analytics to all pages, set GATEWAY_EXTRA_SCRIPTS\n * these scripts will also be run on the pages served by cellxgene server via the --scripts parameter\n * See https://github.com/chanzuckerberg/cellxgene/pull/680 for details on --scripts parameter\n\nCurrently we use a bash script that copies the gateway to a \"build\" directory before modifying templates with sed and the like. There is probably a better way.\n\n# Development\n\nWe\u2019re actively developing. Please see the \"future work\" section of the [wiki](https://github.com/Novartis/cellxgene-gateway/wiki#future-work). If you\u2019re interested in being a contributor please reach out to [@alokito](https://github.com/alokito).\n\n## Developer Install\n\nIf you want to develop the code, you will need to clone the repo. Make sure you have the prequesite listed above, then:\n\n1. Clone the repo\n\n```bash\n git clone https://github.com/Novartis/cellxgene-gateway.git\n cd cellxgene-gateway\n```\n\n2. Install requirements with\n\n```bash\npip install -r requirements.txt\n```\n\n3. Install the gateway in developer mode\n\n```bash\npython setup.py develop\n```\n\nFor convenience, the code repo includes a `run.sh.example` shell script to run the gateway.\n\n4. Install pre-commit hooks\n\n```bash\nconda install -c conda-forge pre-commit\npre-commit install\n```\n\n\n## Running Tests\n\n[](https://travis-ci.org/Novartis/cellxgene-gateway)\n\n```bash\n python -m unittest discover tests\n```\n\n## Code Coverage\n```bash\n coverage run -m unittest discover tests\n coverage html\n```\n\n## Running Linters\n\npip install isort flake8 black\n\n```bash\nisort -rc . # rc means recursive, and was deprecated in dev version of isort\nblack .\n```\n\n# Getting Help\n\nIf you need help for any reason, please make a github ticket. One of the contributors should help you out.\n\n# Releasing New Versions\n\n## How to prepare for release\n\n- Update Changelog.md and version number in __init__.py\n- Cut a release on github\n\t- Go to your project homepage on GitHub\n\t- On right side, you will see [Releases](https://github.com/Novartis/cellxgene-gateway/releases) link. Click on it.\n\t- Click on Draft a new release\n\t- Fill in all the details\n\t\t- Tag version should be the version number of your package release\n\t\t- Release Title can be anything you want, but we use v0.3.11 (the same as the tag to be created on publish)\n\t\t- Description should be changelog\n\t- Click Publish release at the bottom of the page\n\t- Now under Releases you can view all of your releases.\n\t- Copy the download link (tar.gz) and save it somewhere\n\n## How to publish to PyPI\n\nMake sure your `.pypirc` is set up for testpypi and pypi index servers.\n \n\n```bash\nrm -rf dist\npython setup.py sdist bdist_wheel\npython -m twine upload --repository testpypi dist/*\npython -m twine upload dist/*\n```\n\n# Contributors\n\n* Niket Patel - https://github.com/NiketPatel9\n* Alok Saldanha - https://github.com/alokito\n* Yohann Potier - https://github.com/ypotier\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Cellxgene Gateway",
"version": "0.4.0",
"project_urls": {
"Homepage": "http://github.com/Novartis/cellxgene-gateway"
},
"split_keywords": [
"visualization",
"genomics"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3db499997290035047ca8191fb3923414549a328f9df0413a0d593ff754b5b3c",
"md5": "956a418539760a3a77ef5974d25bfc2a",
"sha256": "298530f043ebf7b9cac3a6dc7944a20b6e39c3f528ce92693e22abf5c70317f4"
},
"downloads": -1,
"filename": "cellxgene_gateway-0.4.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "956a418539760a3a77ef5974d25bfc2a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 63456,
"upload_time": "2024-03-10T13:37:07",
"upload_time_iso_8601": "2024-03-10T13:37:07.796412Z",
"url": "https://files.pythonhosted.org/packages/3d/b4/99997290035047ca8191fb3923414549a328f9df0413a0d593ff754b5b3c/cellxgene_gateway-0.4.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "af45d065124ee1800d00db0ab126721f816082f22b873f37f315d40c29fc9e35",
"md5": "0ddf17194d18aab6584c1f4eb5430309",
"sha256": "c218ecf29dd7127acad02dca9bbfb9f776ec1df5b8a2384024d83cd1e2a77485"
},
"downloads": -1,
"filename": "cellxgene-gateway-0.4.0.tar.gz",
"has_sig": false,
"md5_digest": "0ddf17194d18aab6584c1f4eb5430309",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 44491,
"upload_time": "2024-03-10T13:37:09",
"upload_time_iso_8601": "2024-03-10T13:37:09.777303Z",
"url": "https://files.pythonhosted.org/packages/af/45/d065124ee1800d00db0ab126721f816082f22b873f37f315d40c29fc9e35/cellxgene-gateway-0.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-10 13:37:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Novartis",
"github_project": "cellxgene-gateway",
"travis_ci": true,
"coveralls": true,
"github_actions": true,
"requirements": [],
"lcname": "cellxgene-gateway"
}
JSON
