# FirecrestExecutor: A supercomputer at your pythonic fingertips
Execute Python functions on a remote supercomputer. Sounds difficult? It is
not!
The package builds on
[FirecREST](https://eth-cscs.github.io/firecrest-v2/#firecrest-v2), a
lightweight REST API for accessing HPC resources and the associated
[pyFirecREST Python library](https://github.com/eth-cscs/pyfirecrest). It
abstracts these resources in the form of a standard [Python
executor](https://docs.python.org/3/library/concurrent.futures.html#executor-objects§).
Once the executor is created, which requires credentials and information about
the remote supercomputer, you can submit Python functions to be executed
remotely. The executor will transparently take care of the details, such as
creating a job script, submitting it to the scheduler, and waiting for the
results.
## A simple example
```python
from firecrest-executor import FirecrestExecutor
import logging
# functions to be executed remotely
def square_number(x):
from math import pow
return pow(x, 2)
def report_hostname():
import subprocess
import os
cluster = os.environ.get("CLUSTER_NAME", "unknown")
hostname = subprocess.check_output(["hostname"], text=True).strip()
return f"Success testing execution on {hostname} of {cluster}!"
print("Executing remotely, stay tuned...")
# Use the executor to remotely execute (asynchronously) the functions defined above.
with FirecrestExecutor(
working_dir="/users/vjoost/FirecrestExecutorDemo/",
sbatch_options=[
"--job-name=FirecrestExecutor",
"--time=00:10:00",
"--nodes=1",
"--partition=normal",
],
srun_options=["--environment=/users/vjoost/FirecrestExecutorDemo/demo.toml"],
sleep_interval=5,
logger_level=logging.ERROR,
) as executor:
# A quick test to see if the executor is working
print(executor.submit(report_hostname).result())
# and let's compute the square of some numbers
numbers = range(2, 5)
print("Let's compute squares of 2..4: ", list(executor.map(square_number, numbers)))
```
This results in the expected output:
```text
$ python simple.py
Executing remotely, stay tuned...
Success testing execution on nid005463 of clariden!
Let's compute squares of 2..4: [4.0, 9.0, 16.0]
```
## Getting started
### Install the firecrest-executor
Directly from PyPI:
```bash
pip install firecrest-executor
```
or clone from github:
```bash
git clone https://github.com/vondele/firecrest-executor.git
cd firecrest-executor
pip install -e .[examples]
```
### Enable FirecREST
This package requires you have access to a supercomputer that supports the
firecREST API. Inquire with your HPC support team if you are unsure. For the
[Swiss national supercomputing center (CSCS)](https://www.cscs.ch/) this is
the case for all clusters see [their
documentation](https://docs.cscs.ch/services/firecrest/). Follow the process to
obtain the necessary clients, tokens, and credentials to be able to access the
system using the firecREST API and define the following environment variables:
```text
- FIRECREST_CLIENT_ID
- FIRECREST_CLIENT_SECRET
- AUTH_TOKEN_URL
- FIRECREST_URL
- FIRECREST_SYSTEM
- FIRECREST_ACCOUNT
```
### Ensure consistent environments
This package requires the same version of Python and of its dependencies (in
particular the package to serialize Python functions and variables
[dill](https://pypi.org/project/dill/)) to be available locally and remotely.
Containers to the rescue (or manage your environment carefully with any other
tool)! At CSCS the [container
engine](https://docs.cscs.ch/software/container-engine/) allows for passing the
`--environment=foo.toml` flag to srun, to start commands in a container with
the specified settings. Hence, use an equivalent container locally and
remotely. This container image should also contain the Python packages needed
by any of the functions you want to execute remotely.
### Code with remote execution in mind
Python functions that will be execute remotely should be serializable with dill
and be executable without extra context in a fresh Python shell. Avoid access
to global variables (in particular things like locks). Explicitly import all
modules and functions that are used in the remote function.
Currently, every function call creates a new job, allocates at least a full
node, and hence is not suitable for very small tasks. The overhead of starting
a job is a few seconds/minutes at least. This approach is thus suitable for
tasks that are computationally intensive.
The function arguments and return values are serialized and passed between the
systems. Currently, these can not exceed a few 100kB. The functions can not
rely on terminal input and output is stored remotely (typically
`slurm-<jobid>.out`). Wrap functions, capture, and return output if needed.
## Advanced usage and caveats
Containers allow for mounting filesystems, and as such these Python functions
can load and save large persistent datasets. The firecrest api allows for up and
downloading data.
The code prioritizes returning a result, even if e.g. node failure might cause
a job to fail. In these cases, the job is transparently resubmitted and the
result is returned when available. If the job modifies state on the remote
system, this might lead to unexpected results. Similarly, the executor handles
failure of the API by repeated calling, after a few seconds sleep.
Exceptions in the remotely executed function are currently not propagated to
the caller.
The executor has a few knobs for configuration, such as the logging level, the
maximum number of concurrent jobs, and the sleep interval between checks for
status. Passing an explicit environment variable, allows for bypassing the OS
environment variables, and e.g. for creating different executors for different
clusters.
Raw data
{
"_id": null,
"home_page": null,
"name": "firecrest-executor",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "HPC, supercomputing, executor, firecrest",
"author": null,
"author_email": "Joost VandeVondele <Joost.VandeVondele@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/39/2a/6a537eefd46470e50835e22dc101bf04fa81befbae2dd4db635305e452b5/firecrest_executor-0.2.0.tar.gz",
"platform": null,
"description": "# FirecrestExecutor: A supercomputer at your pythonic fingertips\n\nExecute Python functions on a remote supercomputer. Sounds difficult? It is\nnot!\n\nThe package builds on\n[FirecREST](https://eth-cscs.github.io/firecrest-v2/#firecrest-v2), a\nlightweight REST API for accessing HPC resources and the associated\n[pyFirecREST Python library](https://github.com/eth-cscs/pyfirecrest). It\nabstracts these resources in the form of a standard [Python\nexecutor](https://docs.python.org/3/library/concurrent.futures.html#executor-objects\u00a7).\nOnce the executor is created, which requires credentials and information about\nthe remote supercomputer, you can submit Python functions to be executed\nremotely. The executor will transparently take care of the details, such as\ncreating a job script, submitting it to the scheduler, and waiting for the\nresults.\n\n## A simple example\n\n```python\nfrom firecrest-executor import FirecrestExecutor\nimport logging\n\n\n# functions to be executed remotely\ndef square_number(x):\n from math import pow\n\n return pow(x, 2)\n\n\ndef report_hostname():\n import subprocess\n import os\n\n cluster = os.environ.get(\"CLUSTER_NAME\", \"unknown\")\n hostname = subprocess.check_output([\"hostname\"], text=True).strip()\n return f\"Success testing execution on {hostname} of {cluster}!\"\n\n\nprint(\"Executing remotely, stay tuned...\")\n\n# Use the executor to remotely execute (asynchronously) the functions defined above.\nwith FirecrestExecutor(\n working_dir=\"/users/vjoost/FirecrestExecutorDemo/\",\n sbatch_options=[\n \"--job-name=FirecrestExecutor\",\n \"--time=00:10:00\",\n \"--nodes=1\",\n \"--partition=normal\",\n ],\n srun_options=[\"--environment=/users/vjoost/FirecrestExecutorDemo/demo.toml\"],\n sleep_interval=5,\n logger_level=logging.ERROR,\n) as executor:\n # A quick test to see if the executor is working\n print(executor.submit(report_hostname).result())\n # and let's compute the square of some numbers\n numbers = range(2, 5)\n print(\"Let's compute squares of 2..4: \", list(executor.map(square_number, numbers)))\n```\n\nThis results in the expected output:\n\n```text\n$ python simple.py \nExecuting remotely, stay tuned...\nSuccess testing execution on nid005463 of clariden!\nLet's compute squares of 2..4: [4.0, 9.0, 16.0]\n```\n\n## Getting started\n\n### Install the firecrest-executor\n\nDirectly from PyPI:\n\n```bash\npip install firecrest-executor\n```\n\nor clone from github:\n\n```bash\ngit clone https://github.com/vondele/firecrest-executor.git\ncd firecrest-executor\npip install -e .[examples]\n```\n\n### Enable FirecREST\n\nThis package requires you have access to a supercomputer that supports the\nfirecREST API. Inquire with your HPC support team if you are unsure. For the\n[Swiss national supercomputing center (CSCS)](https://www.cscs.ch/) this is\nthe case for all clusters see [their\ndocumentation](https://docs.cscs.ch/services/firecrest/). Follow the process to\nobtain the necessary clients, tokens, and credentials to be able to access the\nsystem using the firecREST API and define the following environment variables:\n\n```text\n - FIRECREST_CLIENT_ID\n - FIRECREST_CLIENT_SECRET\n - AUTH_TOKEN_URL\n - FIRECREST_URL\n - FIRECREST_SYSTEM\n - FIRECREST_ACCOUNT\n```\n\n### Ensure consistent environments\n\nThis package requires the same version of Python and of its dependencies (in\nparticular the package to serialize Python functions and variables\n[dill](https://pypi.org/project/dill/)) to be available locally and remotely.\nContainers to the rescue (or manage your environment carefully with any other\ntool)! At CSCS the [container\nengine](https://docs.cscs.ch/software/container-engine/) allows for passing the\n`--environment=foo.toml` flag to srun, to start commands in a container with\nthe specified settings. Hence, use an equivalent container locally and\nremotely. This container image should also contain the Python packages needed\nby any of the functions you want to execute remotely.\n\n### Code with remote execution in mind\n\nPython functions that will be execute remotely should be serializable with dill\nand be executable without extra context in a fresh Python shell. Avoid access\nto global variables (in particular things like locks). Explicitly import all\nmodules and functions that are used in the remote function.\n\nCurrently, every function call creates a new job, allocates at least a full\nnode, and hence is not suitable for very small tasks. The overhead of starting\na job is a few seconds/minutes at least. This approach is thus suitable for\ntasks that are computationally intensive.\n\nThe function arguments and return values are serialized and passed between the\nsystems. Currently, these can not exceed a few 100kB. The functions can not\nrely on terminal input and output is stored remotely (typically\n`slurm-<jobid>.out`). Wrap functions, capture, and return output if needed.\n\n## Advanced usage and caveats\n\nContainers allow for mounting filesystems, and as such these Python functions\ncan load and save large persistent datasets. The firecrest api allows for up and\ndownloading data.\n\nThe code prioritizes returning a result, even if e.g. node failure might cause\na job to fail. In these cases, the job is transparently resubmitted and the\nresult is returned when available. If the job modifies state on the remote\nsystem, this might lead to unexpected results. Similarly, the executor handles\nfailure of the API by repeated calling, after a few seconds sleep.\n\nExceptions in the remotely executed function are currently not propagated to\nthe caller.\n\nThe executor has a few knobs for configuration, such as the logging level, the\nmaximum number of concurrent jobs, and the sleep interval between checks for\nstatus. Passing an explicit environment variable, allows for bypassing the OS\nenvironment variables, and e.g. for creating different executors for different\nclusters.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Execute python functions on a remote supercomputer",
"version": "0.2.0",
"project_urls": {
"Homepage": "https://github.com/vondele/firecrest-executor",
"Repository": "https://github.com/vondele/firecrest-executor"
},
"split_keywords": [
"hpc",
" supercomputing",
" executor",
" firecrest"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "6c36a646664de1ad59ff26504e04b59bf06ffa64baaf86a1e02cb80c93616c2a",
"md5": "6923c99d08bac106d09da481388f8e79",
"sha256": "9c5fba73b61b931b69a5435fb2b5479d051eeae215f2614f5bde5d9a299c9048"
},
"downloads": -1,
"filename": "firecrest_executor-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6923c99d08bac106d09da481388f8e79",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 13613,
"upload_time": "2025-07-20T22:23:36",
"upload_time_iso_8601": "2025-07-20T22:23:36.088339Z",
"url": "https://files.pythonhosted.org/packages/6c/36/a646664de1ad59ff26504e04b59bf06ffa64baaf86a1e02cb80c93616c2a/firecrest_executor-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "392a6a537eefd46470e50835e22dc101bf04fa81befbae2dd4db635305e452b5",
"md5": "0a6feceff49b0beface6230b53600f6e",
"sha256": "ee27fdf31bb192cf8ca1711f87a9694846b37414dd175e174a4c213cf9a47701"
},
"downloads": -1,
"filename": "firecrest_executor-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "0a6feceff49b0beface6230b53600f6e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 13114,
"upload_time": "2025-07-20T22:23:37",
"upload_time_iso_8601": "2025-07-20T22:23:37.882082Z",
"url": "https://files.pythonhosted.org/packages/39/2a/6a537eefd46470e50835e22dc101bf04fa81befbae2dd4db635305e452b5/firecrest_executor-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-20 22:23:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "vondele",
"github_project": "firecrest-executor",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "firecrest-executor"
}
JSON
