| Name | snakemake-executor-plugin-kueue JSON |
| Version |
0.1.0
JSON |
| download |
| home_page | |
| Summary | |
| upload_time | 2024-01-03 06:16:55 |
| maintainer | |
| docs_url | None |
| author | vsoch |
| requires_python | >=3.11,<4.0 |
| license | |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Snakemake Executor Kueue
This is a [snakemake executor plugin](https://github.com/snakemake/snakemake-executor-plugin-interface/)
that enables interaction with [Kueue](https://kueue.sigs.k8s.io/docs/overview/). The plugin will
install Python dependencies that are needed, and it's assumed that you have [installed Kueue and have queues configured](https://kueue.sigs.k8s.io/docs/tasks/run_jobs/#before-you-begin).
**under development** note that the base container is a custom build under my namespace (`vanessa`)
that has clones from main branches (as opposed to releases).
## Usage
### Setup
You will need to create a cluster first. For local development we recommend [kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installing-from-source):
```bash
kind create cluster
```
#### Install Kueue
You will then need to [install Kueue](https://kueue.sigs.k8s.io/docs/installation/) and
[create your local queues](https://kueue.sigs.k8s.io/docs/tasks/administer_cluster_quotas/) with cluster quotas
E.g., here is an example:
```bash
VERSION=v0.4.0
kubectl apply -f https://github.com/kubernetes-sigs/kueue/releases/download/$VERSION/manifests.yaml
```
Then (wait a few minutes until the jobset controller is running.) and:
```bash
kubectl apply -f example/cluster-queue.yaml
kubectl apply -f example/resource-flavor.yaml
kubectl apply -f example/user-queue.yaml
```
```console
clusterqueue.kueue.x-k8s.io/cluster-queue created
resourceflavor.kueue.x-k8s.io/default-flavor created
localqueue.kueue.x-k8s.io/user-queue created
```
You'll also need kubernetes python installed, and of course Snakemake! Assuming you have snakemake and the plugin here installed, you should be good to go.
Here is how I setup a local or development environment.
```bash
python -m venv env
source env/bin/activate
pip install .
```
### Container
Note that while Snakemake still has a lot of moving pieces, the default container is built from the [Dockerfile](Dockerfile) here and provided as `vanessa/snakemake:kueue` in the executor code. Next go into an [example](example) directory to test out the Kueue executor.
### Job Resources
#### Operator
By default, Kueue will use a batchv1/Job for each step. However, you can
customize this to a different operator with the job [resources](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#resources)
via the kueue_operator attribute:
```yaml
rule a:
input: ...
output: ...
resources:
kueue_operator=flux-operator
shell:
"..."
```
Along with a standard batchv1 Job, We currently support the following `operator`s:
- flux-operator: deploy using the [Flux Operator](https://github.com/flux-framework/flux-operator)
- mpi-operator: deploy using the [MPI Operator](https://github.com/kubeflow/mpi-operator/)
- job: (the default) either unset, or set to "job"
Note that you are in charge of installing and configuring the various operators on your cluster!
See the [Kueue tasks](https://kueue.sigs.k8s.io/docs/tasks/) for more details.
#### Container
You can customize the container you are using, which should have minimally Snakemake and your application
software. We have prepared a container with Flux, Snakemake, and the various other plugins for you to get started.
The [Dockerfile](Dockerfile) is packaged here if you'd like to tweak it, e.g.,
```yaml
rule hello_world:
output:
"...",
resources:
container="ghcr.io/rse-ops/mamba:snakemake",
kueue_operator="job"
shell:
"..."
```
#### Memory
The memory defined for Kubernetes is in a string format, and while we could ideally
do a conversion for now we are lazy and ask you to define it directly:
```yaml
rule a:
input: ...
output: ...
resources:
kueue_memory=200M1
shell:
"..."
```
#### Tasks
The Flux Operator can handle tasks for MPI, so you can set them as follows:
```yaml
rule a:
input: ...
output: ...
resources:
kueue_tasks=1
shell:
"..."
```
For examples, check out the [example](example) directory.
## Want to write a plugin?
If you are interested in writing your own plugin, instructions are provided via the [snakemake-executor-plugin-interface](https://github.com/snakemake/snakemake-executor-plugin-interface).
## License
HPCIC DevTools is distributed under the terms of the MIT license.
All new contributions must be made under this license.
See [LICENSE](https://github.com/converged-computing/cloud-select/blob/main/LICENSE),
[COPYRIGHT](https://github.com/converged-computing/cloud-select/blob/main/COPYRIGHT), and
[NOTICE](https://github.com/converged-computing/cloud-select/blob/main/NOTICE) for details.
SPDX-License-Identifier: (MIT)
LLNL-CODE- 842614
Raw data
{
"_id": null,
"home_page": "",
"name": "snakemake-executor-plugin-kueue",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.11,<4.0",
"maintainer_email": "",
"keywords": "",
"author": "vsoch",
"author_email": "vsoch@users.noreply.github.com",
"download_url": "https://files.pythonhosted.org/packages/ef/af/98c803a8ff20e23e610219036df91ab23b93f1180d3257a0c255787a85db/snakemake_executor_plugin_kueue-0.1.0.tar.gz",
"platform": null,
"description": "# Snakemake Executor Kueue\n\nThis is a [snakemake executor plugin](https://github.com/snakemake/snakemake-executor-plugin-interface/)\nthat enables interaction with [Kueue](https://kueue.sigs.k8s.io/docs/overview/). The plugin will\ninstall Python dependencies that are needed, and it's assumed that you have [installed Kueue and have queues configured](https://kueue.sigs.k8s.io/docs/tasks/run_jobs/#before-you-begin).\n\n**under development** note that the base container is a custom build under my namespace (`vanessa`)\nthat has clones from main branches (as opposed to releases).\n\n## Usage\n\n### Setup\n\nYou will need to create a cluster first. For local development we recommend [kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installing-from-source):\n\n```bash\nkind create cluster\n```\n\n#### Install Kueue\n\nYou will then need to [install Kueue](https://kueue.sigs.k8s.io/docs/installation/) and\n[create your local queues](https://kueue.sigs.k8s.io/docs/tasks/administer_cluster_quotas/) with cluster quotas\n\nE.g., here is an example:\n\n```bash\nVERSION=v0.4.0\nkubectl apply -f https://github.com/kubernetes-sigs/kueue/releases/download/$VERSION/manifests.yaml\n```\n\nThen (wait a few minutes until the jobset controller is running.) and:\n\n```bash\nkubectl apply -f example/cluster-queue.yaml \nkubectl apply -f example/resource-flavor.yaml \nkubectl apply -f example/user-queue.yaml \n```\n```console\nclusterqueue.kueue.x-k8s.io/cluster-queue created\nresourceflavor.kueue.x-k8s.io/default-flavor created\nlocalqueue.kueue.x-k8s.io/user-queue created\n```\n\nYou'll also need kubernetes python installed, and of course Snakemake! Assuming you have snakemake and the plugin here installed, you should be good to go.\nHere is how I setup a local or development environment.\n\n```bash\npython -m venv env\nsource env/bin/activate\npip install .\n```\n\n### Container\n\nNote that while Snakemake still has a lot of moving pieces, the default container is built from the [Dockerfile](Dockerfile) here and provided as `vanessa/snakemake:kueue` in the executor code. Next go into an [example](example) directory to test out the Kueue executor.\n\n### Job Resources\n\n#### Operator\n\nBy default, Kueue will use a batchv1/Job for each step. However, you can\ncustomize this to a different operator with the job [resources](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#resources)\nvia the kueue_operator attribute:\n\n```yaml\nrule a:\n input: ...\n output: ...\n resources:\n kueue_operator=flux-operator\n shell:\n \"...\"\n```\n\nAlong with a standard batchv1 Job, We currently support the following `operator`s:\n\n - flux-operator: deploy using the [Flux Operator](https://github.com/flux-framework/flux-operator)\n - mpi-operator: deploy using the [MPI Operator](https://github.com/kubeflow/mpi-operator/)\n - job: (the default) either unset, or set to \"job\"\n\nNote that you are in charge of installing and configuring the various operators on your cluster!\nSee the [Kueue tasks](https://kueue.sigs.k8s.io/docs/tasks/) for more details.\n\n#### Container\n\nYou can customize the container you are using, which should have minimally Snakemake and your application\nsoftware. We have prepared a container with Flux, Snakemake, and the various other plugins for you to get started.\nThe [Dockerfile](Dockerfile) is packaged here if you'd like to tweak it, e.g.,\n\n```yaml\nrule hello_world:\n output:\n \"...\",\n resources: \n container=\"ghcr.io/rse-ops/mamba:snakemake\",\n kueue_operator=\"job\"\n shell:\n \"...\"\n```\n\n#### Memory\n\nThe memory defined for Kubernetes is in a string format, and while we could ideally\ndo a conversion for now we are lazy and ask you to define it directly:\n\n```yaml\nrule a:\n input: ...\n output: ...\n resources:\n kueue_memory=200M1\n shell:\n \"...\"\n```\n\n#### Tasks\n\nThe Flux Operator can handle tasks for MPI, so you can set them as follows:\n\n```yaml\nrule a:\n input: ...\n output: ...\n resources:\n kueue_tasks=1\n shell:\n \"...\"\n```\n\n\nFor examples, check out the [example](example) directory.\n\n## Want to write a plugin?\n\nIf you are interested in writing your own plugin, instructions are provided via the [snakemake-executor-plugin-interface](https://github.com/snakemake/snakemake-executor-plugin-interface).\n\n## License\n\nHPCIC DevTools is distributed under the terms of the MIT license.\nAll new contributions must be made under this license.\n\nSee [LICENSE](https://github.com/converged-computing/cloud-select/blob/main/LICENSE),\n[COPYRIGHT](https://github.com/converged-computing/cloud-select/blob/main/COPYRIGHT), and\n[NOTICE](https://github.com/converged-computing/cloud-select/blob/main/NOTICE) for details.\n\nSPDX-License-Identifier: (MIT)\n\nLLNL-CODE- 842614",
"bugtrack_url": null,
"license": "",
"summary": "",
"version": "0.1.0",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5a2fa20b2bdcc989013ebac543f77961455fa375227c1c32e1c8223954343b49",
"md5": "ef3a27ef406c1c58eb116cb4d7e29766",
"sha256": "3fcf1bffbd8cb12e7654e52634c9020b5b547852ed6011133b4b6c23bae71d5d"
},
"downloads": -1,
"filename": "snakemake_executor_plugin_kueue-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ef3a27ef406c1c58eb116cb4d7e29766",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11,<4.0",
"size": 12249,
"upload_time": "2024-01-03T06:16:53",
"upload_time_iso_8601": "2024-01-03T06:16:53.188984Z",
"url": "https://files.pythonhosted.org/packages/5a/2f/a20b2bdcc989013ebac543f77961455fa375227c1c32e1c8223954343b49/snakemake_executor_plugin_kueue-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "efaf98c803a8ff20e23e610219036df91ab23b93f1180d3257a0c255787a85db",
"md5": "62675a2792a7c3e4e82857e788fcedd3",
"sha256": "cc5b565a155cae0c4dfbbd3a815f5dd0973bd3b8e3eb9a3179415df82bfe0fe0"
},
"downloads": -1,
"filename": "snakemake_executor_plugin_kueue-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "62675a2792a7c3e4e82857e788fcedd3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11,<4.0",
"size": 11934,
"upload_time": "2024-01-03T06:16:55",
"upload_time_iso_8601": "2024-01-03T06:16:55.035333Z",
"url": "https://files.pythonhosted.org/packages/ef/af/98c803a8ff20e23e610219036df91ab23b93f1180d3257a0c255787a85db/snakemake_executor_plugin_kueue-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-03 06:16:55",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "snakemake-executor-plugin-kueue"
}
