# RADICAL-Pilot (RP)
[](https://github.com/radical-cybertools/radical.pilot/actions/workflows/ci.yml)
[](http://radicalpilot.readthedocs.io/en/stable/?badge=stable)
[](https://codecov.io/gh/radical-cybertools/radical.pilot)
[](https://www.bestpractices.dev/projects/8224)
RADICAL-Pilot (RP) executes heterogeneous tasks with maximum concurrency and at
scale. RP can concurrently execute up to $10^5$ heterogeneous tasks, including
single/multi core/GPU and MPI/OpenMP. Tasks can be stand-alone executables or
Python functions and both types of task can be concurrently executed.
RP is a [Pilot system](https://doi.org/10.1145/3177851), i.e., it separates
resource acquisition from using those resources to execute application tasks. RP
acquires resources by submitting a job to an HPC platform, and it can directly
schedule and launch computational tasks on those resources. Thus, tasks are
directly scheduled on the acquired resources, not via the batch system of the
HPC platform. RP supports concurrently using single/multiple pilots on
single/multiple
[high performance computing (HPC) platforms](https://radicalpilot.readthedocs.io/en/stable/supported.html).
RP is written in Python and exposes a simple yet powerful
[API](https://radicalpilot.readthedocs.io/en/stable/apidoc.html). In 15 lines of
code, you can execute an arbitrary number of executables with maximum
concurrency on a
[Linux container](https://hub.docker.com/u/radicalcybertools)
or, by changing `resource`, on one of the
[supported HPC platforms](https://radicalpilot.readthedocs.io/en/stable/supported.html).
```python
import radical.pilot as rp
# Create a session
session = rp.Session()
# Create a pilot manager and a pilot
pmgr = rp.PilotManager(session=session)
pd_init = {'resource': 'local.localhost',
'runtime' : 30,
'cores' : 4}
pdesc = rp.PilotDescription(pd_init)
pilot = pmgr.submit_pilots(pdesc)
# Crate a task manager and describe your tasks
tmgr = rp.TaskManager(session=session)
tmgr.add_pilots(pilot)
tds = list()
for i in range(8):
td = rp.TaskDescription()
td.executable = 'sleep'
td.arguments = ['10']
tds.append(td)
# Submit your tasks for execution
tmgr.submit_tasks(tds)
tmgr.wait_tasks()
# Close your session
session.close(cleanup=True)
```
## Quick Start
Run RP's [quick start tutorial](https://mybinder.org/v2/gh/radical-cybertools/radical.pilot/HEAD?labpath=docs%2Fsource%2Fgetting_started.ipynb) directly on Binder. No installation needed.
After going through the tutorial, install RP and start to code your application:
```shell
python -m venv ~/.ve/radical-pilot
. ~/.ve/radical-pilot/bin/activate
pip install radical.pilot
```
Note that other than `venv`, you can also use
[`virtualenv`](https://radicalpilot.readthedocs.io/en/stable/getting_started.html#Virtualenv),
[`conda`](https://radicalpilot.readthedocs.io/en/stable/getting_started.html#Conda)
or
[`spack`](https://radicalpilot.readthedocs.io/en/stable/getting_started.html#Spack).
For some inspiration, see our RP application
[examples](https://github.com/radical-cybertools/radical.pilot/tree/devel/examples),
starting from
[00_getting_started.py](https://github.com/radical-cybertools/radical.pilot/blob/devel/examples/00_getting_started.py)
.
## Documentation
[RP user documentation](https://radicalpilot.readthedocs.io/en/stable/) uses Sphinx, and it is published on Read the Docs.
[RP tutorials](https://mybinder.org/v2/gh/radical-cybertools/radical.pilot/HEAD) can be run via Binder.
## Developers
RP development uses Git and
[GitHub](https://github.com/radical-cybertools/radical.pilot). RP **requires**
Python3, a virtual environment and a GNU/Linux OS. Clone, install and
test RP:
```shell
python -m venv ~/.ve/rp-docs
. ~/.ve/rp-docs/bin/activate
git clone git@github.com:radical-cybertools/radical.pilot.git
cd radical.pilot
pip install -r requirements-docs.txt
sphinx-build -M html docs/source/ docs/build/
```
RP documentation uses tutorials coded as Jupyter notebooks. `Sphinx` and
`nbsphinx` run RP locally to execute those tutorials. Successful compilation of
the documentation also serves as a validation of your local development
environment.
## Provide Feedback
Have a question, feature request or you found a bug? Feel free to open a
[support ticket](https://github.com/radical-cybertools/radical.pilot/issues).
For vulnerabilities, please draft a private
[security advisory](https://github.com/radical-cybertools/radical.pilot/security/advisories).
## Contributing
We welcome everyone that wants to contribute to RP development. We are an open
and welcoming community, committed to making participation a harassment-free
experience for everyone. See our
[Code of Conduct](https://radicalpilot.readthedocs.io/en/stable/process/code_of_conduct.html),
relevant
[technical documentation](https://radicalpilot.readthedocs.io/en/stable/process/contributing.html)
and feel free to
[get in touch](https://github.com/radical-cybertools/radical.pilot/issues).
Raw data
{
"_id": null,
"home_page": "http://radical-cybertools.github.io/radical.pilot/",
"name": "radical.pilot",
"maintainer": "The RADICAL Group",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "radical@rutgers.edu",
"keywords": "radical pilot job",
"author": "RADICAL Group at Rutgers University",
"author_email": "radical@rutgers.edu",
"download_url": "https://files.pythonhosted.org/packages/ee/4b/c0e8a6acdb3ec8f30caa6cb133649a742128b920fdd9bf88b7428c1ab4ff/radical_pilot-1.52.1.tar.gz",
"platform": null,
"description": "# RADICAL-Pilot (RP)\n\n[](https://github.com/radical-cybertools/radical.pilot/actions/workflows/ci.yml)\n[](http://radicalpilot.readthedocs.io/en/stable/?badge=stable)\n[](https://codecov.io/gh/radical-cybertools/radical.pilot)\n[](https://www.bestpractices.dev/projects/8224)\n\nRADICAL-Pilot (RP) executes heterogeneous tasks with maximum concurrency and at\nscale. RP can concurrently execute up to $10^5$ heterogeneous tasks, including\nsingle/multi core/GPU and MPI/OpenMP. Tasks can be stand-alone executables or\nPython functions and both types of task can be concurrently executed.\n\nRP is a [Pilot system](https://doi.org/10.1145/3177851), i.e., it separates\nresource acquisition from using those resources to execute application tasks. RP\nacquires resources by submitting a job to an HPC platform, and it can directly\nschedule and launch computational tasks on those resources. Thus, tasks are\ndirectly scheduled on the acquired resources, not via the batch system of the\nHPC platform. RP supports concurrently using single/multiple pilots on\nsingle/multiple\n[high performance computing (HPC) platforms](https://radicalpilot.readthedocs.io/en/stable/supported.html).\n\nRP is written in Python and exposes a simple yet powerful\n[API](https://radicalpilot.readthedocs.io/en/stable/apidoc.html). In 15 lines of\ncode, you can execute an arbitrary number of executables with maximum\nconcurrency on a\n[Linux container](https://hub.docker.com/u/radicalcybertools)\nor, by changing `resource`, on one of the\n[supported HPC platforms](https://radicalpilot.readthedocs.io/en/stable/supported.html).\n\n```python\nimport radical.pilot as rp\n\n# Create a session\nsession = rp.Session()\n\n# Create a pilot manager and a pilot\npmgr = rp.PilotManager(session=session)\npd_init = {'resource': 'local.localhost',\n 'runtime' : 30,\n 'cores' : 4}\npdesc = rp.PilotDescription(pd_init)\npilot = pmgr.submit_pilots(pdesc)\n\n# Crate a task manager and describe your tasks\ntmgr = rp.TaskManager(session=session)\ntmgr.add_pilots(pilot)\ntds = list()\nfor i in range(8):\n td = rp.TaskDescription()\n td.executable = 'sleep'\n td.arguments = ['10']\n tds.append(td)\n\n# Submit your tasks for execution\ntmgr.submit_tasks(tds)\ntmgr.wait_tasks()\n\n# Close your session\nsession.close(cleanup=True)\n```\n\n## Quick Start\n\nRun RP's [quick start tutorial](https://mybinder.org/v2/gh/radical-cybertools/radical.pilot/HEAD?labpath=docs%2Fsource%2Fgetting_started.ipynb) directly on Binder. No installation needed.\n\nAfter going through the tutorial, install RP and start to code your application:\n\n```shell\npython -m venv ~/.ve/radical-pilot\n. ~/.ve/radical-pilot/bin/activate\npip install radical.pilot\n```\n\nNote that other than `venv`, you can also use\n[`virtualenv`](https://radicalpilot.readthedocs.io/en/stable/getting_started.html#Virtualenv),\n[`conda`](https://radicalpilot.readthedocs.io/en/stable/getting_started.html#Conda)\nor\n[`spack`](https://radicalpilot.readthedocs.io/en/stable/getting_started.html#Spack). \n\nFor some inspiration, see our RP application \n[examples](https://github.com/radical-cybertools/radical.pilot/tree/devel/examples),\nstarting from\n[00_getting_started.py](https://github.com/radical-cybertools/radical.pilot/blob/devel/examples/00_getting_started.py)\n.\n\n## Documentation\n\n[RP user documentation](https://radicalpilot.readthedocs.io/en/stable/) uses Sphinx, and it is published on Read the Docs.\n\n[RP tutorials](https://mybinder.org/v2/gh/radical-cybertools/radical.pilot/HEAD) can be run via Binder.\n\n## Developers\n\nRP development uses Git and\n[GitHub](https://github.com/radical-cybertools/radical.pilot). RP **requires**\nPython3, a virtual environment and a GNU/Linux OS. Clone, install and\ntest RP:\n\n```shell\npython -m venv ~/.ve/rp-docs\n. ~/.ve/rp-docs/bin/activate\ngit clone git@github.com:radical-cybertools/radical.pilot.git\ncd radical.pilot\npip install -r requirements-docs.txt\nsphinx-build -M html docs/source/ docs/build/\n```\n\nRP documentation uses tutorials coded as Jupyter notebooks. `Sphinx` and\n`nbsphinx` run RP locally to execute those tutorials. Successful compilation of\nthe documentation also serves as a validation of your local development\nenvironment.\n\n## Provide Feedback\n\nHave a question, feature request or you found a bug? Feel free to open a \n[support ticket](https://github.com/radical-cybertools/radical.pilot/issues).\nFor vulnerabilities, please draft a private \n[security advisory](https://github.com/radical-cybertools/radical.pilot/security/advisories).\n\n## Contributing\n\nWe welcome everyone that wants to contribute to RP development. We are an open\nand welcoming community, committed to making participation a harassment-free\nexperience for everyone. See our\n[Code of Conduct](https://radicalpilot.readthedocs.io/en/stable/process/code_of_conduct.html),\nrelevant\n[technical documentation](https://radicalpilot.readthedocs.io/en/stable/process/contributing.html)\nand feel free to \n[get in touch](https://github.com/radical-cybertools/radical.pilot/issues).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "The RADICAL pilot job framework",
"version": "1.52.1",
"project_urls": {
"Documentation": "https://radicalshare/radical.pilot.readthedocs.io/en/latest/",
"Homepage": "http://radical-cybertools.github.io/radical.pilot/",
"Issues": "https://github.com/radical-cybertools/radical.pilot/issues",
"Source": "https://github.com/radical-cybertools/radical.pilot/"
},
"split_keywords": [
"radical",
"pilot",
"job"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ee4bc0e8a6acdb3ec8f30caa6cb133649a742128b920fdd9bf88b7428c1ab4ff",
"md5": "a3333fb0b519748aad7b5ac7fd8b4121",
"sha256": "ad477f68032e8a52b7c78bfa69557ebb8aba3ed69347590ae77620fe59114052"
},
"downloads": -1,
"filename": "radical_pilot-1.52.1.tar.gz",
"has_sig": false,
"md5_digest": "a3333fb0b519748aad7b5ac7fd8b4121",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 484147,
"upload_time": "2024-04-18T05:18:15",
"upload_time_iso_8601": "2024-04-18T05:18:15.378533Z",
"url": "https://files.pythonhosted.org/packages/ee/4b/c0e8a6acdb3ec8f30caa6cb133649a742128b920fdd9bf88b7428c1ab4ff/radical_pilot-1.52.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-18 05:18:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "radical-cybertools",
"github_project": "radical.pilot",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "radical.pilot"
}
JSON
