# ST4SD Runtime Core
This repository contains the runtime-core of the [Simulation Toolkit for Scientific Discovery (ST4SD)](https://github.ibm.com/st4sd/overview).
The ST4SD-Runtime is a python framework, and associated services, for creating and deploying virtual-experiments - data-flows which embody the measurement of properties of systems.
A data-flow is a workflow which allows consumers to run concurrently with their producers if desired.
Developers describe their data-flows using a YAML configuration file, which is interpreted and executed by the ST4SD-Runtime.
ST4SD-Runtime supports multiple execution-backends including Kubernetes and LSF and a single YAML file can support multiple-archs and multiple run-modes via overlays.
The ST4SD-Runtime also interacts with the [ST4SD-Datastore](https://github.com/st4sd/st4sd-datastore), a database which allows querying of executed virtual-experiments and retrieval of their data.
There are three parts to the ST4SD-Runtime
- `st4sd-runtime-core`: The core python framework (this git repository) for describing and executing virtual experiments
- [`st4sd-runtime-k8s`](https://github.com/st4sd/st4sd-runtime-k8s): Extensions which enable to running and managing virtual-experiments on k8s clusters
- [`st4sd-runtime-service`](https://github.com/st4sd/st4sd-runtime-service): A RESTapi based service allowing users to add, start, stop and query virtual-experiments
## Features
* Cross-platform data-flows
* Supports multiple backends (LSF, OpenShift/Kubernetes, local)
* Abstracts differences between backends allowing a single component description to be used
* Variables can be used to encapsulate platform specific options
* Can define component and platform specific environments
* Co-processing model
* Consumers can be configured to run repeatedly while their producers are alive
* Simple to replicate workflow sub-graphs over sets of inputs
* Supports `do-while` constructs
* Handles task persistence across backend allocation windows and allows user customisable restarts
* Deploy workflows directly from github (Kubernetes stack)
* Store and retrieve data and metadata from st4sd-datastore
## Lightning Start
If you have
1. `python3` with `virtualenv`
2. Have `ssh` access to GitHub set up
The following snippet will install `st4sd-runtime-core` and run a toy-workflow on your laptop
```bash
virtualenv -p python3 $HOME/st4sd-runtime-test
source $HOME/st4sd-runtime-test/bin/activate
pip install st4sd-runtime-core[deploy]
git clone http://github.com/st4sd/sum-numbers.git
elaunch.py --nostamp -l40 sum-numbers
```
This will create a new virtualenv called `st4sd-runtime-test` at `$HOME/st4sd-runtime-test` and install `st4sd-runtime-core` into it. It will also clone a repository into a directory called `sum-numbers` in whatever directory you run the above commands in. It will then run a toy-workflow that takes a couple of minutes to run. The toy workflow output will be in a directory called `sum-numbers.instance`
You can learn more about the toy-workflow, and workflow specification, [here.](https://github.com/st4sd/sum-numbers)
## References
If you use ST4SD in your projects, please consider citing the following:
```bibtex
@software{st4sd_2022,
author = {Johnston, Michael A. and Vassiliadis, Vassilis and Pomponio, Alessandro and Pyzer-Knapp, Edward},
license = {Apache-2.0},
month = {12},
title = {{Simulation Toolkit for Scientific Discovery}},
url = {https://github.com/st4sd/st4sd-runtime-core},
year = {2022}
}
```
## More Information
Our [documentation website](https://st4sd.github.io/overview) contains detailed information on installing ST4SD,
writing and running virtual-experiments, along with much more.
Raw data
{
"_id": null,
"home_page": "https://github.com/st4sd/st4sd-runtime-core",
"name": "st4sd-runtime-core",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "hpc kubernetes openshift lsf workflows experiments computational-chemistry simulation science",
"author": "Michael A. Johnston",
"author_email": "michaelj@ie.ibm.com",
"download_url": "https://files.pythonhosted.org/packages/6d/8c/8452033c25f1860663ca98f981124b020115b12fd79cb0e7007827cd1e74/st4sd_runtime_core-2.2.0.tar.gz",
"platform": null,
"description": "# ST4SD Runtime Core\n\nThis repository contains the runtime-core of the [Simulation Toolkit for Scientific Discovery (ST4SD)](https://github.ibm.com/st4sd/overview).\nThe ST4SD-Runtime is a python framework, and associated services, for creating and deploying virtual-experiments - data-flows which embody the measurement of properties of systems.\n\nA data-flow is a workflow which allows consumers to run concurrently with their producers if desired. \n\nDevelopers describe their data-flows using a YAML configuration file, which is interpreted and executed by the ST4SD-Runtime.\n\nST4SD-Runtime supports multiple execution-backends including Kubernetes and LSF and a single YAML file can support multiple-archs and multiple run-modes via overlays.\n\nThe ST4SD-Runtime also interacts with the [ST4SD-Datastore](https://github.com/st4sd/st4sd-datastore), a database which allows querying of executed virtual-experiments and retrieval of their data.\n\nThere are three parts to the ST4SD-Runtime\n- `st4sd-runtime-core`: The core python framework (this git repository) for describing and executing virtual experiments\n- [`st4sd-runtime-k8s`](https://github.com/st4sd/st4sd-runtime-k8s): Extensions which enable to running and managing virtual-experiments on k8s clusters \n- [`st4sd-runtime-service`](https://github.com/st4sd/st4sd-runtime-service): A RESTapi based service allowing users to add, start, stop and query virtual-experiments\n\n## Features\n\n* Cross-platform data-flows\n * Supports multiple backends (LSF, OpenShift/Kubernetes, local)\n * Abstracts differences between backends allowing a single component description to be used\n * Variables can be used to encapsulate platform specific options\n * Can define component and platform specific environments\n* Co-processing model\n * Consumers can be configured to run repeatedly while their producers are alive\n* Simple to replicate workflow sub-graphs over sets of inputs \n* Supports `do-while` constructs\n* Handles task persistence across backend allocation windows and allows user customisable restarts\n* Deploy workflows directly from github (Kubernetes stack)\n* Store and retrieve data and metadata from st4sd-datastore\n\n## Lightning Start\n\nIf you have \n\n1. `python3` with `virtualenv` \n2. Have `ssh` access to GitHub set up\n\nThe following snippet will install `st4sd-runtime-core` and run a toy-workflow on your laptop\n\n```bash\nvirtualenv -p python3 $HOME/st4sd-runtime-test\nsource $HOME/st4sd-runtime-test/bin/activate\npip install st4sd-runtime-core[deploy]\ngit clone http://github.com/st4sd/sum-numbers.git\nelaunch.py --nostamp -l40 sum-numbers\n```\n\nThis will create a new virtualenv called `st4sd-runtime-test` at `$HOME/st4sd-runtime-test` and install `st4sd-runtime-core` into it. It will also clone a repository into a directory called `sum-numbers` in whatever directory you run the above commands in. It will then run a toy-workflow that takes a couple of minutes to run. The toy workflow output will be in a directory called `sum-numbers.instance`\n\nYou can learn more about the toy-workflow, and workflow specification, [here.](https://github.com/st4sd/sum-numbers) \n\n## References\n\nIf you use ST4SD in your projects, please consider citing the following:\n\n```bibtex\n@software{st4sd_2022,\nauthor = {Johnston, Michael A. and Vassiliadis, Vassilis and Pomponio, Alessandro and Pyzer-Knapp, Edward},\nlicense = {Apache-2.0},\nmonth = {12},\ntitle = {{Simulation Toolkit for Scientific Discovery}},\nurl = {https://github.com/st4sd/st4sd-runtime-core},\nyear = {2022}\n}\n```\n\n## More Information\n\nOur [documentation website](https://st4sd.github.io/overview) contains detailed information on installing ST4SD, \nwriting and running virtual-experiments, along with much more. \n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "A tool for creating and deploying computational experiments",
"version": "2.2.0",
"project_urls": {
"Homepage": "https://github.com/st4sd/st4sd-runtime-core"
},
"split_keywords": [
"hpc",
"kubernetes",
"openshift",
"lsf",
"workflows",
"experiments",
"computational-chemistry",
"simulation",
"science"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "417820846b152135f4ae97faf465deba797d1ff0af0b40be263e7e43943f0298",
"md5": "5e41d2fc18b58b4c515ba2feb57e5642",
"sha256": "ad3cdaacdaa9b236c6ad0668307d6533c889509834f0c41354fdf4c40124302c"
},
"downloads": -1,
"filename": "st4sd_runtime_core-2.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5e41d2fc18b58b4c515ba2feb57e5642",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 698376,
"upload_time": "2024-10-08T15:00:48",
"upload_time_iso_8601": "2024-10-08T15:00:48.403785Z",
"url": "https://files.pythonhosted.org/packages/41/78/20846b152135f4ae97faf465deba797d1ff0af0b40be263e7e43943f0298/st4sd_runtime_core-2.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6d8c8452033c25f1860663ca98f981124b020115b12fd79cb0e7007827cd1e74",
"md5": "533c5ac962caa97449045050da65ba77",
"sha256": "e8f8d5e5e3a70bf6f90f16c49b16f605a44b255b0f38b039513d77e9438e2ea8"
},
"downloads": -1,
"filename": "st4sd_runtime_core-2.2.0.tar.gz",
"has_sig": false,
"md5_digest": "533c5ac962caa97449045050da65ba77",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 745876,
"upload_time": "2024-10-08T15:00:49",
"upload_time_iso_8601": "2024-10-08T15:00:49.893067Z",
"url": "https://files.pythonhosted.org/packages/6d/8c/8452033c25f1860663ca98f981124b020115b12fd79cb0e7007827cd1e74/st4sd_runtime_core-2.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-08 15:00:49",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "st4sd",
"github_project": "st4sd-runtime-core",
"travis_ci": true,
"coveralls": false,
"github_actions": false,
"tox": true,
"lcname": "st4sd-runtime-core"
}
JSON
