# ST4SD Runtime Core
This repository contains the runtime-core of the [Simulation Toolkit for Scientific Discovery (ST4SD)](https://github.ibm.com/st4sd/overview).
The ST4SD-Runtime is a python framework, and associated services, for creating and deploying virtual-experiments - data-flows which embody the measurement of properties of systems.
A data-flow is a workflow which allows consumers to run concurrently with their producers if desired.
Developers describe their data-flows using a YAML configuration file, which is interpreted and executed by the ST4SD-Runtime.
ST4SD-Runtime supports multiple execution-backends including Kubernetes and LSF and a single YAML file can support multiple-archs and multiple run-modes via overlays.
The ST4SD-Runtime also interacts with the [ST4SD-Datastore](https://github.com/st4sd/st4sd-datastore), a database which allows querying of executed virtual-experiments and retrieval of their data.
There are three parts to the ST4SD-Runtime
- `st4sd-runtime-core`: The core python framework (this git repository) for describing and executing virtual experiments
- [`st4sd-runtime-k8s`](https://github.com/st4sd/st4sd-runtime-k8s): Extensions which enable to running and managing virtual-experiments on k8s clusters
- [`st4sd-runtime-service`](https://github.com/st4sd/st4sd-runtime-service): A RESTapi based service allowing users to add, start, stop and query virtual-experiments
## Features
* Cross-platform data-flows
* Supports multiple backends (LSF, OpenShift/Kubernetes, local)
* Abstracts differences between backends allowing a single component description to be used
* Variables can be used to encapsulate platform specific options
* Can define component and platform specific environments
* Co-processing model
* Consumers can be configured to run repeatedly while their producers are alive
* Simple to replicate workflow sub-graphs over sets of inputs
* Supports `do-while` constructs
* Handles task persistence across backend allocation windows and allows user customisable restarts
* Deploy workflows directly from github (Kubernetes stack)
* Store and retrieve data and metadata from st4sd-datastore
## Lightning Start
If you have
1. `python3` with `virtualenv`
2. Have `ssh` access to GitHub set up
The following snippet will install `st4sd-runtime-core` and run a toy-workflow on your laptop
```bash
virtualenv -p python3 $HOME/st4sd-runtime-test
source $HOME/st4sd-runtime-test/bin/activate
pip install st4sd-runtime-core[deploy]
git clone http://github.com/st4sd/sum-numbers.git
elaunch.py --nostamp -l40 sum-numbers
```
This will create a new virtualenv called `st4sd-runtime-test` at `$HOME/st4sd-runtime-test` and install `st4sd-runtime-core` into it. It will also clone a repository into a directory called `sum-numbers` in whatever directory you run the above commands in. It will then run a toy-workflow that takes a couple of minutes to run. The toy workflow output will be in a directory called `sum-numbers.instance`
You can learn more about the toy-workflow, and workflow specification, [here.](https://github.com/st4sd/sum-numbers)
## References
If you use ST4SD in your projects, please consider citing the following:
```bibtex
@software{st4sd_2022,
author = {Johnston, Michael A. and Vassiliadis, Vassilis and Pomponio, Alessandro and Pyzer-Knapp, Edward},
license = {Apache-2.0},
month = {12},
title = {{Simulation Toolkit for Scientific Discovery}},
url = {https://github.com/st4sd/st4sd-runtime-core},
year = {2022}
}
```
## More Information
Our [documentation website](https://st4sd.github.io/overview) contains detailed information on installing ST4SD,
writing and running virtual-experiments, along with much more.
Raw data
{
"_id": null,
"home_page": "https://github.com/st4sd/st4sd-runtime-core",
"name": "st4sd-runtime-core",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "hpc kubernetes openshift lsf workflows experiments computational-chemistry simulation science",
"author": "Michael A. Johnston",
"author_email": "michaelj@ie.ibm.com",
"download_url": "https://files.pythonhosted.org/packages/e7/35/9cf4594c44139e5cd06846392858c19df6ba5ed90c81aee424e56f0eeee8/st4sd_runtime_core-2.5.1.tar.gz",
"platform": null,
"description": "# ST4SD Runtime Core\n\nThis repository contains the runtime-core of the [Simulation Toolkit for Scientific Discovery (ST4SD)](https://github.ibm.com/st4sd/overview).\nThe ST4SD-Runtime is a python framework, and associated services, for creating and deploying virtual-experiments - data-flows which embody the measurement of properties of systems.\n\nA data-flow is a workflow which allows consumers to run concurrently with their producers if desired. \n\nDevelopers describe their data-flows using a YAML configuration file, which is interpreted and executed by the ST4SD-Runtime.\n\nST4SD-Runtime supports multiple execution-backends including Kubernetes and LSF and a single YAML file can support multiple-archs and multiple run-modes via overlays.\n\nThe ST4SD-Runtime also interacts with the [ST4SD-Datastore](https://github.com/st4sd/st4sd-datastore), a database which allows querying of executed virtual-experiments and retrieval of their data.\n\nThere are three parts to the ST4SD-Runtime\n- `st4sd-runtime-core`: The core python framework (this git repository) for describing and executing virtual experiments\n- [`st4sd-runtime-k8s`](https://github.com/st4sd/st4sd-runtime-k8s): Extensions which enable to running and managing virtual-experiments on k8s clusters \n- [`st4sd-runtime-service`](https://github.com/st4sd/st4sd-runtime-service): A RESTapi based service allowing users to add, start, stop and query virtual-experiments\n\n## Features\n\n* Cross-platform data-flows\n * Supports multiple backends (LSF, OpenShift/Kubernetes, local)\n * Abstracts differences between backends allowing a single component description to be used\n * Variables can be used to encapsulate platform specific options\n * Can define component and platform specific environments\n* Co-processing model\n * Consumers can be configured to run repeatedly while their producers are alive\n* Simple to replicate workflow sub-graphs over sets of inputs \n* Supports `do-while` constructs\n* Handles task persistence across backend allocation windows and allows user customisable restarts\n* Deploy workflows directly from github (Kubernetes stack)\n* Store and retrieve data and metadata from st4sd-datastore\n\n## Lightning Start\n\nIf you have \n\n1. `python3` with `virtualenv` \n2. Have `ssh` access to GitHub set up\n\nThe following snippet will install `st4sd-runtime-core` and run a toy-workflow on your laptop\n\n```bash\nvirtualenv -p python3 $HOME/st4sd-runtime-test\nsource $HOME/st4sd-runtime-test/bin/activate\npip install st4sd-runtime-core[deploy]\ngit clone http://github.com/st4sd/sum-numbers.git\nelaunch.py --nostamp -l40 sum-numbers\n```\n\nThis will create a new virtualenv called `st4sd-runtime-test` at `$HOME/st4sd-runtime-test` and install `st4sd-runtime-core` into it. It will also clone a repository into a directory called `sum-numbers` in whatever directory you run the above commands in. It will then run a toy-workflow that takes a couple of minutes to run. The toy workflow output will be in a directory called `sum-numbers.instance`\n\nYou can learn more about the toy-workflow, and workflow specification, [here.](https://github.com/st4sd/sum-numbers) \n\n## References\n\nIf you use ST4SD in your projects, please consider citing the following:\n\n```bibtex\n@software{st4sd_2022,\nauthor = {Johnston, Michael A. and Vassiliadis, Vassilis and Pomponio, Alessandro and Pyzer-Knapp, Edward},\nlicense = {Apache-2.0},\nmonth = {12},\ntitle = {{Simulation Toolkit for Scientific Discovery}},\nurl = {https://github.com/st4sd/st4sd-runtime-core},\nyear = {2022}\n}\n```\n\n## More Information\n\nOur [documentation website](https://st4sd.github.io/overview) contains detailed information on installing ST4SD, \nwriting and running virtual-experiments, along with much more. \n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "A tool for creating and deploying computational experiments",
"version": "2.5.1",
"project_urls": {
"Homepage": "https://github.com/st4sd/st4sd-runtime-core"
},
"split_keywords": [
"hpc",
"kubernetes",
"openshift",
"lsf",
"workflows",
"experiments",
"computational-chemistry",
"simulation",
"science"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "dccf219b367e24f5b2139fe0417b1a867336aacd6cc598851c189410c35315ea",
"md5": "713aaea5b38a52270c125734baca2e7f",
"sha256": "ae53bc16e552f9d94e159574deef5653e2ad5ecb2f9b033f3b04cd467ab01482"
},
"downloads": -1,
"filename": "st4sd_runtime_core-2.5.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "713aaea5b38a52270c125734baca2e7f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 700530,
"upload_time": "2025-04-02T20:24:17",
"upload_time_iso_8601": "2025-04-02T20:24:17.392256Z",
"url": "https://files.pythonhosted.org/packages/dc/cf/219b367e24f5b2139fe0417b1a867336aacd6cc598851c189410c35315ea/st4sd_runtime_core-2.5.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "e7359cf4594c44139e5cd06846392858c19df6ba5ed90c81aee424e56f0eeee8",
"md5": "93b70456e331bce70ddc138fac5dacb9",
"sha256": "342196d389cc255dde1032c90a81542dd0a7e395b24267fecfd807a8ca72632b"
},
"downloads": -1,
"filename": "st4sd_runtime_core-2.5.1.tar.gz",
"has_sig": false,
"md5_digest": "93b70456e331bce70ddc138fac5dacb9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 747403,
"upload_time": "2025-04-02T20:24:19",
"upload_time_iso_8601": "2025-04-02T20:24:19.194187Z",
"url": "https://files.pythonhosted.org/packages/e7/35/9cf4594c44139e5cd06846392858c19df6ba5ed90c81aee424e56f0eeee8/st4sd_runtime_core-2.5.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-04-02 20:24:19",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "st4sd",
"github_project": "st4sd-runtime-core",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"tox": true,
"lcname": "st4sd-runtime-core"
}
JSON
