# HTCdaskGateway
* A Dask Gateway client extension for heterogeneous cluster mode combining the Kubernetes backend for pain-free scheduler networking, with COFFEA-powered HTCondor workers and/or OKD [coming soon].
* Latest [](https://badge.fury.io/py/htcdaskgateway)
is installed by default and deployed to the COFFEA-DASK notebook on EAF (https://analytics-hub.fnal.gov). A few lines will get you going!
* The current image for workers/schedulers is: coffeateam/coffea-dask-cc7-gateway:0.7.12-fastjet-3.3.4.0rc9-g8a990fa
## Basic usage @ Fermilab [EAF](https://analytics-hub.fnal.gov)
* Make sure the notebook launched supports this functionality (COFFEA-DASK notebook)
```
from htcdaskgateway import HTCGateway
gateway = HTCGateway()
cluster = gateway.new_cluster()
cluster
# Scale my cluster to 5 HTCondor workers
cluster.scale(5)
# Obtain a client for connecting to your cluster scheduler
# Your cluster should be ready to take requests
client = cluster.get_client()
client
# When computations are finished, shutdown the cluster
cluster.shutdown()
```
## Other functions worth checking out
* This is a multi-tenant environment, and you are authenticated via JupyterHub Oauth which means that you can create as many* clusters as you wish
* To list your clusters:
```
# Verify that the gateway is responding to requests by asking to list all its clusters
clusters = gateway.list_clusters()
clusters
```
* To connect to a specific cluster from the list:
```
cluster = gateway.connect(cluster_name)
cluster
cluster.shutdown()
```
* To gracefully close the cluster and remove HTCondor worker jobs associated to it:
```
cluster.shutdown()
```
* There are widgets implemented by Dask Gateway. Make sure to give them a try from your EAF COFFEA notebook, just execute the the `client` and `cluster` commands (after properly initializing them) in a cell like:
```
-------------
cluster = gateway.new_cluster()
cluster
< Widget will appear after this step>
-------------
client = cluster.get_client()
client
< Widget will apear after this step >
-------------
cluster
< Widget will appear after this step >
```
Raw data
{
"_id": null,
"home_page": "https://github.com/mapsacosta/htcdaskgateway",
"name": "htcdaskgateway",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Maria P. Acosta F./Fermilab EAF project",
"author_email": "macosta@fnal.gov",
"download_url": "https://files.pythonhosted.org/packages/5a/46/a456c00a3e87e23fb7ee4bea0070ddc61498574a0ff2810f3b276e08e4d9/htcdaskgateway-0.1.22.tar.gz",
"platform": null,
"description": "# HTCdaskGateway\n\n* A Dask Gateway client extension for heterogeneous cluster mode combining the Kubernetes backend for pain-free scheduler networking, with COFFEA-powered HTCondor workers and/or OKD [coming soon].\n* Latest [](https://badge.fury.io/py/htcdaskgateway)\n is installed by default and deployed to the COFFEA-DASK notebook on EAF (https://analytics-hub.fnal.gov). A few lines will get you going!\n* The current image for workers/schedulers is: coffeateam/coffea-dask-cc7-gateway:0.7.12-fastjet-3.3.4.0rc9-g8a990fa\n\n## Basic usage @ Fermilab [EAF](https://analytics-hub.fnal.gov)\n* Make sure the notebook launched supports this functionality (COFFEA-DASK notebook)\n\n```\nfrom htcdaskgateway import HTCGateway\n\ngateway = HTCGateway()\ncluster = gateway.new_cluster()\ncluster\n\n# Scale my cluster to 5 HTCondor workers\ncluster.scale(5)\n\n# Obtain a client for connecting to your cluster scheduler\n# Your cluster should be ready to take requests\nclient = cluster.get_client()\nclient\n\n# When computations are finished, shutdown the cluster\ncluster.shutdown()\n```\n## Other functions worth checking out\n* This is a multi-tenant environment, and you are authenticated via JupyterHub Oauth which means that you can create as many* clusters as you wish\n* To list your clusters:\n```\n# Verify that the gateway is responding to requests by asking to list all its clusters\nclusters = gateway.list_clusters()\nclusters\n```\n* To connect to a specific cluster from the list:\n```\ncluster = gateway.connect(cluster_name)\ncluster\ncluster.shutdown()\n```\n* To gracefully close the cluster and remove HTCondor worker jobs associated to it:\n```\ncluster.shutdown()\n```\n* There are widgets implemented by Dask Gateway. Make sure to give them a try from your EAF COFFEA notebook, just execute the the `client` and `cluster` commands (after properly initializing them) in a cell like: \n```\n-------------\ncluster = gateway.new_cluster()\ncluster\n< Widget will appear after this step>\n-------------\nclient = cluster.get_client()\nclient\n< Widget will apear after this step >\n-------------\ncluster\n< Widget will appear after this step >\n```\n",
"bugtrack_url": null,
"license": "Apache",
"summary": "Launches a Dask Gateway cluster in K8s and joins HTCondor workers to it",
"version": "0.1.22",
"project_urls": {
"Homepage": "https://github.com/mapsacosta/htcdaskgateway"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "857940048e19b0176f85c7ebdd29552c341ea932e31d54dc0c2b73afce0cf699",
"md5": "3d242c77f122aa26436e16e344934524",
"sha256": "a3d55c2f7f83ef5c86320af4ddcb98f4b2010718e195c184d822201d1b548bfd"
},
"downloads": -1,
"filename": "htcdaskgateway-0.1.22-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3d242c77f122aa26436e16e344934524",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 10895,
"upload_time": "2024-12-05T19:24:43",
"upload_time_iso_8601": "2024-12-05T19:24:43.978958Z",
"url": "https://files.pythonhosted.org/packages/85/79/40048e19b0176f85c7ebdd29552c341ea932e31d54dc0c2b73afce0cf699/htcdaskgateway-0.1.22-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5a46a456c00a3e87e23fb7ee4bea0070ddc61498574a0ff2810f3b276e08e4d9",
"md5": "7722f5ee308a0f9fa4717e44dc5ae61c",
"sha256": "f23347ad83d8e97e204b25c252fadc8f5b78acfc579a44a10756a07d53635614"
},
"downloads": -1,
"filename": "htcdaskgateway-0.1.22.tar.gz",
"has_sig": false,
"md5_digest": "7722f5ee308a0f9fa4717e44dc5ae61c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 7495,
"upload_time": "2024-12-05T19:24:47",
"upload_time_iso_8601": "2024-12-05T19:24:47.331201Z",
"url": "https://files.pythonhosted.org/packages/5a/46/a456c00a3e87e23fb7ee4bea0070ddc61498574a0ff2810f3b276e08e4d9/htcdaskgateway-0.1.22.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-05 19:24:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mapsacosta",
"github_project": "htcdaskgateway",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "htcdaskgateway"
}
JSON
