# ST4SD Runtime Core
This repository contains the runtime-core of the [Simulation Toolkit for Scientific Discovery (ST4SD)](https://github.ibm.com/st4sd/overview).
The ST4SD-Runtime is a python framework, and associated services, for creating and deploying virtual-experiments - data-flows which embody the measurement of properties of systems.
A data-flow is a workflow which allows consumers to run concurrently with their producers if desired.
Developers describe their data-flows using a YAML configuration file, which is interpreted and executed by the ST4SD-Runtime.
ST4SD-Runtime supports multiple execution-backends including Kubernetes and LSF and a single YAML file can support multiple-archs and multiple run-modes via overlays.
The ST4SD-Runtime also interacts with the [ST4SD-Datastore](https://github.com/st4sd/st4sd-datastore), a database which allows querying of executed virtual-experiments and retrieval of their data.
There are three parts to the ST4SD-Runtime
- `st4sd-runtime-core`: The core python framework (this git repository) for describing and executing virtual experiments
- [`st4sd-runtime-k8s`](https://github.com/st4sd/st4sd-runtime-k8s): Extensions which enable to running and managing virtual-experiments on k8s clusters
- [`st4sd-runtime-service`](https://github.com/st4sd/st4sd-runtime-service): A RESTapi based service allowing users to add, start, stop and query virtual-experiments
## Features
* Cross-platform data-flows
* Supports multiple backends (LSF, OpenShift/Kubernetes, local)
* Abstracts differences between backends allowing a single component description to be used
* Variables can be used to encapsulate platform specific options
* Can define component and platform specific environments
* Co-processing model
* Consumers can be configured to run repeatedly while their producers are alive
* Simple to replicate workflow sub-graphs over sets of inputs
* Supports `do-while` constructs
* Handles task persistence across backend allocation windows and allows user customisable restarts
* Deploy workflows directly from github (Kubernetes stack)
* Store and retrieve data and metadata from st4sd-datastore
## Lightning Start
If you have
1. `python3` with `virtualenv`
2. Have `ssh` access to GitHub set up
The following snippet will install `st4sd-runtime-core` and run a toy-workflow on your laptop
```bash
virtualenv -p python3 $HOME/st4sd-runtime-test
source $HOME/st4sd-runtime-test/bin/activate
pip install st4sd-runtime-core[deploy]
git clone http://github.com/st4sd/sum-numbers.git
elaunch.py --nostamp -l40 sum-numbers
```
This will create a new virtualenv called `st4sd-runtime-test` at `$HOME/st4sd-runtime-test` and install `st4sd-runtime-core` into it. It will also clone a repository into a directory called `sum-numbers` in whatever directory you run the above commands in. It will then run a toy-workflow that takes a couple of minutes to run. The toy workflow output will be in a directory called `sum-numbers.instance`
You can learn more about the toy-workflow, and workflow specification, [here.](https://github.com/st4sd/sum-numbers)
## References
If you use ST4SD in your projects, please consider citing the following:
```bibtex
@software{st4sd_2022,
author = {Johnston, Michael A. and Vassiliadis, Vassilis and Pomponio, Alessandro and Pyzer-Knapp, Edward},
license = {Apache-2.0},
month = {12},
title = {{Simulation Toolkit for Scientific Discovery}},
url = {https://github.com/st4sd/st4sd-runtime-core},
year = {2022}
}
```
## More Information
Our [documentation website](https://st4sd.github.io/overview) contains detailed information on installing ST4SD,
writing and running virtual-experiments, along with much more.
Raw data
{
"_id": null,
"home_page": "https://github.com/st4sd/st4sd-runtime-core",
"name": "st4sd-runtime-core",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "hpc kubernetes openshift lsf workflows experiments computational-chemistry simulation science",
"author": "Michael A. Johnston",
"author_email": "michaelj@ie.ibm.com",
"download_url": "https://files.pythonhosted.org/packages/cd/4e/241932a9194e991e5d4332e2b293a6e6939b29cbaf8ee80012ae0a98cb82/st4sd-runtime-core-2.1.2.tar.gz",
"platform": null,
"description": "# ST4SD Runtime Core\n\nThis repository contains the runtime-core of the [Simulation Toolkit for Scientific Discovery (ST4SD)](https://github.ibm.com/st4sd/overview).\nThe ST4SD-Runtime is a python framework, and associated services, for creating and deploying virtual-experiments - data-flows which embody the measurement of properties of systems.\n\nA data-flow is a workflow which allows consumers to run concurrently with their producers if desired. \n\nDevelopers describe their data-flows using a YAML configuration file, which is interpreted and executed by the ST4SD-Runtime.\n\nST4SD-Runtime supports multiple execution-backends including Kubernetes and LSF and a single YAML file can support multiple-archs and multiple run-modes via overlays.\n\nThe ST4SD-Runtime also interacts with the [ST4SD-Datastore](https://github.com/st4sd/st4sd-datastore), a database which allows querying of executed virtual-experiments and retrieval of their data.\n\nThere are three parts to the ST4SD-Runtime\n- `st4sd-runtime-core`: The core python framework (this git repository) for describing and executing virtual experiments\n- [`st4sd-runtime-k8s`](https://github.com/st4sd/st4sd-runtime-k8s): Extensions which enable to running and managing virtual-experiments on k8s clusters \n- [`st4sd-runtime-service`](https://github.com/st4sd/st4sd-runtime-service): A RESTapi based service allowing users to add, start, stop and query virtual-experiments\n\n## Features\n\n* Cross-platform data-flows\n * Supports multiple backends (LSF, OpenShift/Kubernetes, local)\n * Abstracts differences between backends allowing a single component description to be used\n * Variables can be used to encapsulate platform specific options\n * Can define component and platform specific environments\n* Co-processing model\n * Consumers can be configured to run repeatedly while their producers are alive\n* Simple to replicate workflow sub-graphs over sets of inputs \n* Supports `do-while` constructs\n* Handles task persistence across backend allocation windows and allows user customisable restarts\n* Deploy workflows directly from github (Kubernetes stack)\n* Store and retrieve data and metadata from st4sd-datastore\n\n## Lightning Start\n\nIf you have \n\n1. `python3` with `virtualenv` \n2. Have `ssh` access to GitHub set up\n\nThe following snippet will install `st4sd-runtime-core` and run a toy-workflow on your laptop\n\n```bash\nvirtualenv -p python3 $HOME/st4sd-runtime-test\nsource $HOME/st4sd-runtime-test/bin/activate\npip install st4sd-runtime-core[deploy]\ngit clone http://github.com/st4sd/sum-numbers.git\nelaunch.py --nostamp -l40 sum-numbers\n```\n\nThis will create a new virtualenv called `st4sd-runtime-test` at `$HOME/st4sd-runtime-test` and install `st4sd-runtime-core` into it. It will also clone a repository into a directory called `sum-numbers` in whatever directory you run the above commands in. It will then run a toy-workflow that takes a couple of minutes to run. The toy workflow output will be in a directory called `sum-numbers.instance`\n\nYou can learn more about the toy-workflow, and workflow specification, [here.](https://github.com/st4sd/sum-numbers) \n\n## References\n\nIf you use ST4SD in your projects, please consider citing the following:\n\n```bibtex\n@software{st4sd_2022,\nauthor = {Johnston, Michael A. and Vassiliadis, Vassilis and Pomponio, Alessandro and Pyzer-Knapp, Edward},\nlicense = {Apache-2.0},\nmonth = {12},\ntitle = {{Simulation Toolkit for Scientific Discovery}},\nurl = {https://github.com/st4sd/st4sd-runtime-core},\nyear = {2022}\n}\n```\n\n## More Information\n\nOur [documentation website](https://st4sd.github.io/overview) contains detailed information on installing ST4SD, \nwriting and running virtual-experiments, along with much more. \n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "A tool for creating and deploying computational experiments",
"version": "2.1.2",
"project_urls": {
"Homepage": "https://github.com/st4sd/st4sd-runtime-core"
},
"split_keywords": [
"hpc",
"kubernetes",
"openshift",
"lsf",
"workflows",
"experiments",
"computational-chemistry",
"simulation",
"science"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9a4e48b846a4452038643a1e20cbe097cd804a8456d37f1d289ca2f68c893a82",
"md5": "ee76f5be03830682bc487fe5c097a1f3",
"sha256": "c7bdd04a413b37513f743cc2e46e1cdc3823057c8d91cb2aa8625de88b8ffb6b"
},
"downloads": -1,
"filename": "st4sd_runtime_core-2.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ee76f5be03830682bc487fe5c097a1f3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 700361,
"upload_time": "2024-01-16T09:17:36",
"upload_time_iso_8601": "2024-01-16T09:17:36.092945Z",
"url": "https://files.pythonhosted.org/packages/9a/4e/48b846a4452038643a1e20cbe097cd804a8456d37f1d289ca2f68c893a82/st4sd_runtime_core-2.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "cd4e241932a9194e991e5d4332e2b293a6e6939b29cbaf8ee80012ae0a98cb82",
"md5": "b2f47046bdec97acf213462bb14d3496",
"sha256": "dc71ea71dfb7cbf9b6b49da832dfb930db01f67039aa8c7c087fa7e61b54a4c4"
},
"downloads": -1,
"filename": "st4sd-runtime-core-2.1.2.tar.gz",
"has_sig": false,
"md5_digest": "b2f47046bdec97acf213462bb14d3496",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 743354,
"upload_time": "2024-01-16T09:17:38",
"upload_time_iso_8601": "2024-01-16T09:17:38.538401Z",
"url": "https://files.pythonhosted.org/packages/cd/4e/241932a9194e991e5d4332e2b293a6e6939b29cbaf8ee80012ae0a98cb82/st4sd-runtime-core-2.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-16 09:17:38",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "st4sd",
"github_project": "st4sd-runtime-core",
"travis_ci": true,
"coveralls": false,
"github_actions": false,
"tox": true,
"lcname": "st4sd-runtime-core"
}
JSON
