miniwdl-lsf
=============
Extends miniwdl to run workflows on LSF clusters in singularity containers.
This [LSF backend](https://miniwdl.readthedocs.io/en/latest/runner_backends.html) plugin for
[miniwdl](https://github.com/chanzuckerberg/miniwdl) runs WDL task containers
by creating a job script that is submitted to a LSF cluster. In case the job
description has a container, singularity will be used as container runtime.
Installation
------------
For the development version::
pip install git+https://github.com/adthrasher/miniwdl-lsf.git
LSF-specific runtime hints
--------------
`miniwdl-lsf` supports LSF-specific runtime hints. These can be specified using the `object` syntax under an `lsf` object.
```
runtime {
lsf: object {
time: 30
}
}
```
Currently supported hints:
- time: passed to LSF's `-W` argument. Sets the runtime limit for a task in minutes. [LSF documentation](https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=o-w-1)
Note: `hints` are in a state of flux within the WDL specification. WDL 1.1 added support for `hints` as part of the `runtime` section. These could be nested using the `object` syntax. However, `object` will be removed in future versions of WDL and the `hints` will become a distinct section in the task definition.
`miniwdl check` may return warnings when attempting to validate WDL tasks with an `lsf` object hint. Similar to the warnings below. These warnings do not affect the `miniwdl run` command.
```
(Ln 41, Col 14) UnknownRuntimeKey, unknown entry in task runtime section: lsf
(Ln 41, Col 14) Deprecated, replace 'object' with specific struct type [WDL >= 1.1]
```
Configuration
--------------
The following [miniwdl configuration](https://miniwdl.readthedocs.io/en/latest/runner_reference.html#configuration)
example can be used to use miniwdl on a LSF cluster:
```
[scheduler]
container_backend=lsf_singularity
# Sets the maximum concurrent tasks. Since LSF handles scheduling, we only
# limit to avoid excessive overhead in miniwdl.
task_concurrency=200
# This setting allows running tasks to continue, even if one other tasks fails.
# Useful in combination with call caching. Prevents wasting resources by
# cancelling jobs half-way that would probably succeed.
fail_fast = false
[call_cache]
# The following settings create a call cache under the current directory.
# This prevents wasting unnecessary resources on the cluster by rerunning
# jobs that have already succeeded.
put = true
get = true
dir = "$PWD/miniwdl_call_cache"
[task_runtime]
# Setting a 'maxRetries' default allows jobs that fail due to intermittent
# errors on the cluster to be retried.
defaults = {
"maxRetries": 2,
"docker": "ubuntu:20.04"
}
command_shell = /bin/bash
memory_limit_multiplier = 1.0
[singularity]
# This plugin wraps the singularity backend. Make sure the settings are
# appropriate for your cluster.
exe = ["singularity"]
# the miniwdl default options contain options to run as a fake root, which
# is not available on most clusters.
run_options = [
"--containall",
"--cleanenv"
]
# Location of the singularity images (optional). The miniwdl-lsf plugin
# will set it to a directory inside $PWD. This location must be reachable
# for the submit nodes.
image_cache = "$PWD/miniwdl_singularity_cache"
[lsf]
# extra arguments passed to the bsub command (optional).
extra_args=""
# Task memory specifications should be interpreted as per-job not per-core (LSF default)
memory_per_job = true
```
Acknowledgements
--------------
`miniwdl-lsf` is originally based on [`miniwdl-slurm`](https://github.com/miniwdl-ext/miniwdl-slurm).
Raw data
{
"_id": null,
"home_page": "https://github.com/adthrasher/miniwdl-lsf/",
"name": "miniwdl-lsf",
"maintainer": "",
"docs_url": null,
"requires_python": ">3.6",
"maintainer_email": "",
"keywords": "WDL,miniwdl,workflow language,LSF,Singularity",
"author": "Andrew Thrasher",
"author_email": "adthrasher@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/d4/3d/dcd3adaf7693e70a7aa496906f3bb7aae030fc551cd2954250f5fb84bc67/miniwdl_lsf-0.2.0.tar.gz",
"platform": null,
"description": "miniwdl-lsf\n=============\nExtends miniwdl to run workflows on LSF clusters in singularity containers.\n\nThis [LSF backend](https://miniwdl.readthedocs.io/en/latest/runner_backends.html) plugin for\n[miniwdl](https://github.com/chanzuckerberg/miniwdl) runs WDL task containers\nby creating a job script that is submitted to a LSF cluster. In case the job\ndescription has a container, singularity will be used as container runtime.\n\nInstallation\n------------\nFor the development version::\n\n pip install git+https://github.com/adthrasher/miniwdl-lsf.git\n\nLSF-specific runtime hints\n--------------\n`miniwdl-lsf` supports LSF-specific runtime hints. These can be specified using the `object` syntax under an `lsf` object.\n\n```\nruntime {\n lsf: object {\n time: 30\n }\n}\n```\n\nCurrently supported hints:\n- time: passed to LSF's `-W` argument. Sets the runtime limit for a task in minutes. [LSF documentation](https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=o-w-1)\n\nNote: `hints` are in a state of flux within the WDL specification. WDL 1.1 added support for `hints` as part of the `runtime` section. These could be nested using the `object` syntax. However, `object` will be removed in future versions of WDL and the `hints` will become a distinct section in the task definition.\n\n`miniwdl check` may return warnings when attempting to validate WDL tasks with an `lsf` object hint. Similar to the warnings below. These warnings do not affect the `miniwdl run` command.\n\n```\n (Ln 41, Col 14) UnknownRuntimeKey, unknown entry in task runtime section: lsf\n (Ln 41, Col 14) Deprecated, replace 'object' with specific struct type [WDL >= 1.1]\n```\n\nConfiguration\n--------------\nThe following [miniwdl configuration](https://miniwdl.readthedocs.io/en/latest/runner_reference.html#configuration)\nexample can be used to use miniwdl on a LSF cluster:\n\n```\n [scheduler]\n container_backend=lsf_singularity\n # Sets the maximum concurrent tasks. Since LSF handles scheduling, we only\n # limit to avoid excessive overhead in miniwdl.\n task_concurrency=200\n \n # This setting allows running tasks to continue, even if one other tasks fails. \n # Useful in combination with call caching. Prevents wasting resources by\n # cancelling jobs half-way that would probably succeed.\n fail_fast = false\n\n [call_cache]\n # The following settings create a call cache under the current directory.\n # This prevents wasting unnecessary resources on the cluster by rerunning \n # jobs that have already succeeded.\n put = true \n get = true \n dir = \"$PWD/miniwdl_call_cache\"\n\n [task_runtime]\n # Setting a 'maxRetries' default allows jobs that fail due to intermittent\n # errors on the cluster to be retried.\n defaults = {\n \"maxRetries\": 2,\n \"docker\": \"ubuntu:20.04\"\n }\n\n command_shell = /bin/bash\n\n memory_limit_multiplier = 1.0\n \n [singularity]\n # This plugin wraps the singularity backend. Make sure the settings are\n # appropriate for your cluster.\n exe = [\"singularity\"]\n\n # the miniwdl default options contain options to run as a fake root, which\n # is not available on most clusters.\n run_options = [\n \"--containall\",\n \"--cleanenv\"\n ]\n\n # Location of the singularity images (optional). The miniwdl-lsf plugin\n # will set it to a directory inside $PWD. This location must be reachable\n # for the submit nodes.\n image_cache = \"$PWD/miniwdl_singularity_cache\"\n\n [lsf]\n # extra arguments passed to the bsub command (optional).\n extra_args=\"\"\n # Task memory specifications should be interpreted as per-job not per-core (LSF default)\n memory_per_job = true\n```\n\nAcknowledgements\n--------------\n`miniwdl-lsf` is originally based on [`miniwdl-slurm`](https://github.com/miniwdl-ext/miniwdl-slurm).",
"bugtrack_url": null,
"license": "MIT",
"summary": "miniwdl lsf backend using singularity",
"version": "0.2.0",
"project_urls": {
"Documentation": "https://adthrasher.github.io/miniwdl-lsf/",
"Homepage": "https://github.com/adthrasher/miniwdl-lsf/",
"Repository": "https://github.com/adthrasher/miniwdl-lsf/"
},
"split_keywords": [
"wdl",
"miniwdl",
"workflow language",
"lsf",
"singularity"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "06cac78acb84d267fbac809877cf59fba07f7e795378b2753496cad015eab0fe",
"md5": "178ca684b0e9e4e4634bf2de51b75c0d",
"sha256": "80e044694a906899fceacde1c2a6ee7ca2dcb9f1dd855aa15c511ec8c4d29986"
},
"downloads": -1,
"filename": "miniwdl_lsf-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "178ca684b0e9e4e4634bf2de51b75c0d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">3.6",
"size": 6369,
"upload_time": "2024-02-14T20:40:14",
"upload_time_iso_8601": "2024-02-14T20:40:14.579134Z",
"url": "https://files.pythonhosted.org/packages/06/ca/c78acb84d267fbac809877cf59fba07f7e795378b2753496cad015eab0fe/miniwdl_lsf-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d43ddcd3adaf7693e70a7aa496906f3bb7aae030fc551cd2954250f5fb84bc67",
"md5": "b3bb52d0ded61566512d3402d109ca00",
"sha256": "c8c79b3c2580e2a9d728b579406a6c5c30fc21a00b71f8f52763253c0eb0d799"
},
"downloads": -1,
"filename": "miniwdl_lsf-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "b3bb52d0ded61566512d3402d109ca00",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">3.6",
"size": 5131,
"upload_time": "2024-02-14T20:40:15",
"upload_time_iso_8601": "2024-02-14T20:40:15.997007Z",
"url": "https://files.pythonhosted.org/packages/d4/3d/dcd3adaf7693e70a7aa496906f3bb7aae030fc551cd2954250f5fb84bc67/miniwdl_lsf-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-14 20:40:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "adthrasher",
"github_project": "miniwdl-lsf",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "miniwdl-lsf"
}
JSON
