# Model Ensembler
[](https://www.repostatus.org/#wip)
[](CODE_OF_CONDUCT.md)
`model-ensembler`is a tool to configure and automate running model ensembles on High-Performance Computing (HPC) clusters.
It reduces the manual configuration of individual ensemble runs, by using a common configuration to individually generate templates
for each run.
It also provides pre-and post processing functionality to allow for common tasks to be applied to the ensemble, before
and after the individual runs.
`model-ensembler` is developed to be extendable to various HPC backends,currently supporting SLURM and running locally.
## Installation
To install `model-ensembler`:
```
python -m venv venv
source venv/bin/activate
pip install model-ensembler
```
To check it has installed correctly, you can run:
```
model_ensemble_check [dummy|slurm]
```
## Basic Usage
Under the `examples/` folder you will find example configs and templates that we can run on a local machine:
```shell
examples/
├── template_job/
│ ├── inputfile.j2
│ ├── pre_run.sh.j2
│ ├── slurm_run.sh.js
│ └── post_run.sh.j2
└── ensemble_config.yml
```
The command `model_ensemble` is provided to execute the ensemble.
The `--help` flag can be used to find out more information:
```shell
model_ensemble --help
```
Its use is as follows:
```shell
model_ensemble configuration {slurm, dummy}
```
Here `configuration` refers to our configuration file, and `{slurm, dummy}`
are the HPC backend options (where `dummy` is the options to run locally).
Applying this to our `examples/`, and running locally:
```shell
model_ensemble examples/sanity-check.yml dummy
```
You are now running a model ensemble!
## Documentation
For further usage instructions and an overview of `model-ensembler`, please refer to the documentation.
## Future plans
Current plans are captured now in the github issues. There's nothing in the
long term that I'm focusing on for this tool, except to maintain it and see
if I can promote the usage a bit more.
This tool was merely to help out with a single support ticket for a weather
model run, but the concept had potential and it was easier than deploying
something more substantial! If there are better approaches or tools that do
something similar, very keen to look at them!
Certainly, things like Airflow and job arrays have similar concepts, but are
either more heavyweight/less suitable deployment wise or not abstracted enough
for simplifying lives, respectively!!!
## Cylc
Compare [Cylc](https://cylc.github.io/) to `model-ensembler`.
## Environmental Forecasting
The `model-ensembler` is part of a wider family of tools for [Environmental Forecasting](https://github.com/environmental-forecasting):
* [download-toolbox](https://github.com/environmental-forecasting/download-toolbox): A toolbox of downloaders for environmental data.
* [preprocess-toolbox](https://github.com/environmental-forecasting/preprocess-toolbox): A toolbox for processing downloaded datasets according to common approaches for environmental data.
## Copyright
[MIT LICENSE](LICENSE)
© British Antarctic Survey 2021-2025
Raw data
{
"_id": null,
"home_page": null,
"name": "model-ensembler",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "James Byrne <jambyr@bas.ac.uk>, Thomas Zwagerman <thozwa@bas.ac.uk>",
"keywords": "slurm, hpc, tools, batch, model, ensemble, local",
"author": null,
"author_email": "James Byrne <jambyr@bas.ac.uk>",
"download_url": "https://files.pythonhosted.org/packages/b4/71/8f2d84d9210422082ad1986d98ad2cdf60408646d9481c7c0c503446df65/model_ensembler-0.6.0.tar.gz",
"platform": null,
"description": "# Model Ensembler\n\n\n\n\n\n[](https://www.repostatus.org/#wip)\n[](CODE_OF_CONDUCT.md)\n\n`model-ensembler`is a tool to configure and automate running model ensembles on High-Performance Computing (HPC) clusters.\nIt reduces the manual configuration of individual ensemble runs, by using a common configuration to individually generate templates\nfor each run.\n\nIt also provides pre-and post processing functionality to allow for common tasks to be applied to the ensemble, before\nand after the individual runs. \n\n`model-ensembler` is developed to be extendable to various HPC backends,currently supporting SLURM and running locally. \n\n## Installation\nTo install `model-ensembler`:\n```\npython -m venv venv\nsource venv/bin/activate\npip install model-ensembler\n```\n\nTo check it has installed correctly, you can run:\n```\nmodel_ensemble_check [dummy|slurm]\n```\n\n## Basic Usage\nUnder the `examples/` folder you will find example configs and templates that we can run on a local machine:\n\n```shell\nexamples/\n\u251c\u2500\u2500 template_job/\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 inputfile.j2\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 pre_run.sh.j2\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 slurm_run.sh.js\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 post_run.sh.j2\n\u2514\u2500\u2500 ensemble_config.yml\n```\n\nThe command `model_ensemble` is provided to execute the ensemble.\n\nThe `--help` flag can be used to find out more information:\n\n```shell\nmodel_ensemble --help\n```\nIts use is as follows:\n\n```shell\nmodel_ensemble configuration {slurm, dummy}\n```\n\nHere `configuration` refers to our configuration file, and `{slurm, dummy}`\nare the HPC backend options (where `dummy` is the options to run locally). \n\nApplying this to our `examples/`, and running locally:\n\n```shell\nmodel_ensemble examples/sanity-check.yml dummy\n```\n\nYou are now running a model ensemble!\n\n## Documentation\nFor further usage instructions and an overview of `model-ensembler`, please refer to the documentation.\n\n## Future plans\nCurrent plans are captured now in the github issues. There's nothing in the \nlong term that I'm focusing on for this tool, except to maintain it and see \nif I can promote the usage a bit more. \n\nThis tool was merely to help out with a single support ticket for a weather \nmodel run, but the concept had potential and it was easier than deploying \nsomething more substantial! If there are better approaches or tools that do \nsomething similar, very keen to look at them! \n\nCertainly, things like Airflow and job arrays have similar concepts, but are \neither more heavyweight/less suitable deployment wise or not abstracted enough \nfor simplifying lives, respectively!!!\n\n## Cylc\n\nCompare [Cylc](https://cylc.github.io/) to `model-ensembler`. \n\n## Environmental Forecasting\nThe `model-ensembler` is part of a wider family of tools for [Environmental Forecasting](https://github.com/environmental-forecasting):\n\n* [download-toolbox](https://github.com/environmental-forecasting/download-toolbox): A toolbox of downloaders for environmental data.\n* [preprocess-toolbox](https://github.com/environmental-forecasting/preprocess-toolbox): A toolbox for processing downloaded datasets according to common approaches for environmental data.\n\n## Copyright\n[MIT LICENSE](LICENSE)\n\n© British Antarctic Survey 2021-2025\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Model Ensembler for managed batch workflows",
"version": "0.6.0",
"project_urls": {
"Bug Tracker": "https://www.github.com/jimcircadian/model-ensembler/issues",
"Change Log": "https://github.com/environmental-forecasting/model-ensembler/blob/main/CHANGELOG.md",
"Homepage": "https://www.github.com/jimcircadian/model-ensembler"
},
"split_keywords": [
"slurm",
" hpc",
" tools",
" batch",
" model",
" ensemble",
" local"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "e78d6461320c799c474af8c2c4284a2eba150a5ae4ab7be043283d806e052703",
"md5": "886ed2b0d5723e94a9d68b29913bdc6f",
"sha256": "74fdb38cd30615afdf058fa0e2bca1f42142384a85f6e2fe9a19e3dbc9d0e244"
},
"downloads": -1,
"filename": "model_ensembler-0.6.0-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "886ed2b0d5723e94a9d68b29913bdc6f",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.7",
"size": 17332,
"upload_time": "2025-07-16T10:13:56",
"upload_time_iso_8601": "2025-07-16T10:13:56.284144Z",
"url": "https://files.pythonhosted.org/packages/e7/8d/6461320c799c474af8c2c4284a2eba150a5ae4ab7be043283d806e052703/model_ensembler-0.6.0-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "b4718f2d84d9210422082ad1986d98ad2cdf60408646d9481c7c0c503446df65",
"md5": "7e355189d0c4ddd262ab2cb871eefcd6",
"sha256": "2ebf6a2bc08265a658286baa40a4c59e47dd820cbec1a08928521aa865e899e4"
},
"downloads": -1,
"filename": "model_ensembler-0.6.0.tar.gz",
"has_sig": false,
"md5_digest": "7e355189d0c4ddd262ab2cb871eefcd6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 21404,
"upload_time": "2025-07-16T10:13:57",
"upload_time_iso_8601": "2025-07-16T10:13:57.096612Z",
"url": "https://files.pythonhosted.org/packages/b4/71/8f2d84d9210422082ad1986d98ad2cdf60408646d9481c7c0c503446df65/model_ensembler-0.6.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-16 10:13:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jimcircadian",
"github_project": "model-ensembler",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "model-ensembler"
}
JSON
