# remotemanager
Modular serialisation and management package for handling the running of functions on remote machines
Based off of the BigDFT RemoteRunner concept, remotemanager represents an improvement and expansion on the concepts based there.
Primary usage is via a `Dataset`, which connects to a remote machine via `URL`
You can think of the `Dataset` as a "container" of sorts for a calculation, to which "runs" are attached. These runs are then executed on the remote machine described by the provided `URL`
## Installation
A quick install of the latest stable release can be done via `pip install remotemanager`
For development, you can clone this repo and install via `cd remotemanager && pip install -e .[dev]`
Tip: You can clone a specific branch with `git clone -b devel`.
If you want to build the docs locally a `pandoc` install is required.
You can install all required python packages with the `[dev]` or `[docs]` optionals.
## HPC
Remotemanager exists to facilitate running on High Performance Compute machines (supercomputers). Script generation is ideally done via the `BaseComputer` module.
Existing Computers can be found at [this repository](https://gitlab.com/l_sim/remotemanager-computers). For creating a new machine class, see the [documentation](https://l_sim.gitlab.io/remotemanager/).
## Documentation
See the [documentation](https://l_sim.gitlab.io/remotemanager/) for further information, tutorials and api documentation.
## Quickstart
This section will run through running a very basic function on a machine.
It roughly echoes the [quickstart](https://l_sim.gitlab.io/remotemanager/tutorials/A1_Quickstart.html) page found in the
docs.
### Function Definition
Start by defining your "calculation" as a python function.
```python
def multiply(a, b):
import time
time.sleep(1)
return a * b
```
### Remote Connection
We need to be able to connect to a remote machine to run this function.
Assuming you can connect to a machine with a string like `ssh user@host` or just `ssh machine`, you can directly create
a connection.
Use the `URL` module for this.
```python
from remotemanager import URL
connection = URL("ssh@machine")
```
### Commands
You can execute commands on this machine using the `cmd` method:
```python
connection.cmd("pwd")
>> > "/home/user"
```
### Running Functions
To execute your function on the specified machine, create a `Dataset`
```python
from remotemanager import Dataset
ds = Dataset(function=multiply, url=connection, name="test")
```
#### Adding Runs
You can specify the inputs that your function should use by adding `Runner` instances to the `Dataset`
```python
ds.append_run({"a": 10, "b": 7})
```
#### Running
Now run your `Dataset` with `run()`.
You can wait for completion with `wait()`
```python
ds.run()
ds.wait(1, 10)
```
Here, we are waiting for a maximum of 10s, and checking for results every 1s
### Results
Result collection is done in two stages.
Once a run is complete, we must first fetch the results with `fetch_results`.
Then we can access the results at the `results` property
```python
ds.fetch_results()
ds.results
>> > [70]
```
## sanzu
The `sanzu` functionality allows you to tag a jupyter cell for remote running.
The cell will be converted into a function, set off to the specified remote machine, and executed there.
For detailed information, see the
relevant [section of the docs](https://l_sim.gitlab.io/remotemanager/tutorials/D2_Jupyter_Magic.html)
To use this functionality, first enable the magic:
```python
%load_ext
remotemanager
```
You should then create a `URL` instance for your machine:
```python
from remotemanager import URL
connection = URL(...)
```
And now we can execute any cell on this machine by using the `%%sanzu` protocol:
```python
%%sanzu
url = connection
%%sargs
a = 10
%%sargs
b = 7
a * b
>> > 70
```
This can be useful for doing file operations on the remote machine, however it is possible to access your results.
### Accessing Sanzu Results
A sanzu run will inject a `magic_dataset` object into the jupyter runtime.
This is the `Dataset` that was used to execute the most recent cell, so you can access the information from there.
For our last run, we can see our results here:
```python
print(magic_dataset.results)
>> > [70]
```
Raw data
{
"_id": null,
"home_page": null,
"name": "remotemanager",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "HPC, remote, scheduler, workflow, workflows, distributed",
"author": null,
"author_email": "Louis Beal <louis.j.beal@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/62/4d/8f54647b781560347ce72be37b687a9f139b32300ccac200d2532df6386c/remotemanager-0.11.9.tar.gz",
"platform": null,
"description": "# remotemanager\n\nModular serialisation and management package for handling the running of functions on remote machines\n\nBased off of the BigDFT RemoteRunner concept, remotemanager represents an improvement and expansion on the concepts based there.\n\nPrimary usage is via a `Dataset`, which connects to a remote machine via `URL`\n\nYou can think of the `Dataset` as a \"container\" of sorts for a calculation, to which \"runs\" are attached. These runs are then executed on the remote machine described by the provided `URL`\n\n## Installation\n\nA quick install of the latest stable release can be done via `pip install remotemanager`\n\nFor development, you can clone this repo and install via `cd remotemanager && pip install -e .[dev]`\n\nTip: You can clone a specific branch with `git clone -b devel`.\n\nIf you want to build the docs locally a `pandoc` install is required.\n\nYou can install all required python packages with the `[dev]` or `[docs]` optionals.\n\n## HPC\n\nRemotemanager exists to facilitate running on High Performance Compute machines (supercomputers). Script generation is ideally done via the `BaseComputer` module.\n\nExisting Computers can be found at [this repository](https://gitlab.com/l_sim/remotemanager-computers). For creating a new machine class, see the [documentation](https://l_sim.gitlab.io/remotemanager/).\n\n## Documentation\n\nSee the [documentation](https://l_sim.gitlab.io/remotemanager/) for further information, tutorials and api documentation.\n\n## Quickstart\n\nThis section will run through running a very basic function on a machine.\n\nIt roughly echoes the [quickstart](https://l_sim.gitlab.io/remotemanager/tutorials/A1_Quickstart.html) page found in the\ndocs.\n\n### Function Definition\n\nStart by defining your \"calculation\" as a python function.\n\n```python\ndef multiply(a, b):\n import time\n\n time.sleep(1)\n\n return a * b\n```\n\n### Remote Connection\n\nWe need to be able to connect to a remote machine to run this function.\n\nAssuming you can connect to a machine with a string like `ssh user@host` or just `ssh machine`, you can directly create\na connection.\n\nUse the `URL` module for this.\n\n```python\nfrom remotemanager import URL\n\nconnection = URL(\"ssh@machine\")\n```\n\n### Commands\n\nYou can execute commands on this machine using the `cmd` method:\n\n```python\nconnection.cmd(\"pwd\")\n>> > \"/home/user\"\n```\n\n### Running Functions\n\nTo execute your function on the specified machine, create a `Dataset`\n\n```python\nfrom remotemanager import Dataset\n\nds = Dataset(function=multiply, url=connection, name=\"test\")\n```\n\n#### Adding Runs\n\nYou can specify the inputs that your function should use by adding `Runner` instances to the `Dataset`\n\n```python\nds.append_run({\"a\": 10, \"b\": 7})\n```\n\n#### Running\n\nNow run your `Dataset` with `run()`.\n\nYou can wait for completion with `wait()`\n\n```python\nds.run()\n\nds.wait(1, 10)\n```\n\nHere, we are waiting for a maximum of 10s, and checking for results every 1s\n\n### Results\n\nResult collection is done in two stages.\n\nOnce a run is complete, we must first fetch the results with `fetch_results`.\n\nThen we can access the results at the `results` property\n\n```python\nds.fetch_results()\n\nds.results\n\n>> > [70]\n```\n\n## sanzu\n\nThe `sanzu` functionality allows you to tag a jupyter cell for remote running.\n\nThe cell will be converted into a function, set off to the specified remote machine, and executed there.\n\nFor detailed information, see the\nrelevant [section of the docs](https://l_sim.gitlab.io/remotemanager/tutorials/D2_Jupyter_Magic.html)\n\nTo use this functionality, first enable the magic:\n\n```python\n%load_ext\nremotemanager\n```\n\nYou should then create a `URL` instance for your machine:\n\n```python\nfrom remotemanager import URL\n\nconnection = URL(...)\n```\n\nAnd now we can execute any cell on this machine by using the `%%sanzu` protocol:\n\n```python\n%%sanzu\nurl = connection\n%%sargs\na = 10\n%%sargs\nb = 7\n\na * b\n\n>> > 70\n```\n\nThis can be useful for doing file operations on the remote machine, however it is possible to access your results.\n\n### Accessing Sanzu Results\n\nA sanzu run will inject a `magic_dataset` object into the jupyter runtime.\n\nThis is the `Dataset` that was used to execute the most recent cell, so you can access the information from there.\n\nFor our last run, we can see our results here:\n\n```python\nprint(magic_dataset.results)\n\n>> > [70]\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "remote run management tool",
"version": "0.11.9",
"project_urls": {
"Changelog": "https://l_sim.gitlab.io/remotemanager/versions.html",
"Documentation": "https://l_sim.gitlab.io/remotemanager/",
"Homepage": "https://gitlab.com/l_sim/remotemanager/",
"Issues": "https://gitlab.com/l_sim/remotemanager/-/issues",
"Repository": "https://gitlab.com/l_sim/remotemanager/"
},
"split_keywords": [
"hpc",
" remote",
" scheduler",
" workflow",
" workflows",
" distributed"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1511660dcb9ca06832d7d63489e468a345e3971037e36849195242337d6fde6a",
"md5": "f86f55e0010b4a116fc8e50b6fb2c9f5",
"sha256": "93fd56a80221aaba85fdb3b6a80046d12b155bc397bfe4798cb2ce56c494e110"
},
"downloads": -1,
"filename": "remotemanager-0.11.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f86f55e0010b4a116fc8e50b6fb2c9f5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 103263,
"upload_time": "2024-05-02T09:08:26",
"upload_time_iso_8601": "2024-05-02T09:08:26.840085Z",
"url": "https://files.pythonhosted.org/packages/15/11/660dcb9ca06832d7d63489e468a345e3971037e36849195242337d6fde6a/remotemanager-0.11.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "624d8f54647b781560347ce72be37b687a9f139b32300ccac200d2532df6386c",
"md5": "1c555f9a2eaf044449edce1d865b62c7",
"sha256": "281c3d9e11ee11b76e63455eedef12ebfe101e1403004c93b11dd7b6ccd3a570"
},
"downloads": -1,
"filename": "remotemanager-0.11.9.tar.gz",
"has_sig": false,
"md5_digest": "1c555f9a2eaf044449edce1d865b62c7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 87357,
"upload_time": "2024-05-02T09:08:30",
"upload_time_iso_8601": "2024-05-02T09:08:30.568833Z",
"url": "https://files.pythonhosted.org/packages/62/4d/8f54647b781560347ce72be37b687a9f139b32300ccac200d2532df6386c/remotemanager-0.11.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-02 09:08:30",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "l_sim",
"gitlab_project": "remotemanager",
"lcname": "remotemanager"
}
JSON