# PyStributed: Remote Execution Utility
This package allows users to mark specific code cells in a Jupyter Notebook for remote execution. Once marked, the code will be packaged into a Docker container, sent to a specified remote server for execution, and the results will be fetched back to the local machine.
## Installation
1. Extract the contents of `pystributed.zip` to a location on your machine.
2. Navigate to the directory and run `pip install .` to install the package.
## Prerequisites
- Docker installed on both local and remote machines.
- SSH access to the remote machine.
- PyTorch and Transformers libraries if you are using them in your code.
## Configuration
Before using the package, you need to set up some configurations in `config.py`:
- `DOCKER_IMAGE_NAME`: Name of the Docker image that will be created.
- `DOCKER_REGISTRY`: Docker registry where the image will be pushed.
- `REMOTE_SERVER`: SSH-compatible address of your remote server (e.g., `user@remote_server_ip`).
- `REMOTE_WORKDIR`: Working directory on the remote server where results will be stored.
- `USER_CODE_PATH`: Temporary path on the local machine where the user's code will be saved before packaging.
## Usage
1. In your Jupyter Notebook, import the package:
```python
import pystributed.main as runner
```
2. Use the `%%save_for_remote` magic command to mark the code cell you want to run remotely:
```python
%%save_for_remote
# Your code here
# For example:
import torch
model = torch.load('my_model.pth')
result = model(some_data)
```
3. After marking the desired code cell, call the main function from the package to execute the process:
```python
runner.main()
```
## Under the Hood
The package works in the following sequence:
1. The code cell marked with `%%save_for_remote` is saved to a Python script (`user_code.py` by default).
2. A Docker image is built with the user's code and necessary dependencies.
3. The Docker image is pushed to the specified Docker registry.
4. The package SSHs into the specified remote server, pulls the Docker image, and runs it.
5. Once the code execution is complete on the remote server, the results are fetched and saved to the local machine.
Raw data
{
"_id": null,
"home_page": "https://github.com/yourusername/pystributed",
"name": "pystributed",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Your Name",
"author_email": "your.email@example.com",
"download_url": "https://files.pythonhosted.org/packages/f4/ae/d8b267e09fdadbbade7cb8dfbfbfe171504c8094308a2506fe477ec3939d/pystributed-0.1.4.tar.gz",
"platform": null,
"description": "\n# PyStributed: Remote Execution Utility\n\nThis package allows users to mark specific code cells in a Jupyter Notebook for remote execution. Once marked, the code will be packaged into a Docker container, sent to a specified remote server for execution, and the results will be fetched back to the local machine.\n\n## Installation\n\n1. Extract the contents of `pystributed.zip` to a location on your machine.\n2. Navigate to the directory and run `pip install .` to install the package.\n\n## Prerequisites\n\n- Docker installed on both local and remote machines.\n- SSH access to the remote machine.\n- PyTorch and Transformers libraries if you are using them in your code.\n\n## Configuration\n\nBefore using the package, you need to set up some configurations in `config.py`:\n\n- `DOCKER_IMAGE_NAME`: Name of the Docker image that will be created.\n- `DOCKER_REGISTRY`: Docker registry where the image will be pushed.\n- `REMOTE_SERVER`: SSH-compatible address of your remote server (e.g., `user@remote_server_ip`).\n- `REMOTE_WORKDIR`: Working directory on the remote server where results will be stored.\n- `USER_CODE_PATH`: Temporary path on the local machine where the user's code will be saved before packaging.\n\n## Usage\n\n1. In your Jupyter Notebook, import the package:\n\n```python\nimport pystributed.main as runner\n```\n\n2. Use the `%%save_for_remote` magic command to mark the code cell you want to run remotely:\n\n```python\n%%save_for_remote\n\n# Your code here\n# For example:\nimport torch\nmodel = torch.load('my_model.pth')\nresult = model(some_data)\n```\n\n3. After marking the desired code cell, call the main function from the package to execute the process:\n\n```python\nrunner.main()\n```\n\n## Under the Hood\n\nThe package works in the following sequence:\n\n1. The code cell marked with `%%save_for_remote` is saved to a Python script (`user_code.py` by default).\n2. A Docker image is built with the user's code and necessary dependencies.\n3. The Docker image is pushed to the specified Docker registry.\n4. The package SSHs into the specified remote server, pulls the Docker image, and runs it.\n5. Once the code execution is complete on the remote server, the results are fetched and saved to the local machine.\n",
"bugtrack_url": null,
"license": "",
"summary": "A utility to run Jupyter Notebook code on a remote server using Docker.",
"version": "0.1.4",
"project_urls": {
"Homepage": "https://github.com/yourusername/pystributed"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "de1db665be47c2090732571f05fd63447db273988f711fae1c763102e5e5c2c4",
"md5": "25b5098c3ce91149e1912910214311ee",
"sha256": "e8801b2bcf9a1ff971dac78ae4d7d5ece0753c430754eedacb70088199b189a0"
},
"downloads": -1,
"filename": "pystributed-0.1.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "25b5098c3ce91149e1912910214311ee",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 5473,
"upload_time": "2023-11-01T16:03:42",
"upload_time_iso_8601": "2023-11-01T16:03:42.167472Z",
"url": "https://files.pythonhosted.org/packages/de/1d/b665be47c2090732571f05fd63447db273988f711fae1c763102e5e5c2c4/pystributed-0.1.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f4aed8b267e09fdadbbade7cb8dfbfbfe171504c8094308a2506fe477ec3939d",
"md5": "8bdb28758ff030aae519ce48f4565124",
"sha256": "07f19c4bccf8c73bdabbae3e29acf509f952936cd878c3e7e126a4fc2d74e90f"
},
"downloads": -1,
"filename": "pystributed-0.1.4.tar.gz",
"has_sig": false,
"md5_digest": "8bdb28758ff030aae519ce48f4565124",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 2968,
"upload_time": "2023-11-01T16:03:44",
"upload_time_iso_8601": "2023-11-01T16:03:44.800400Z",
"url": "https://files.pythonhosted.org/packages/f4/ae/d8b267e09fdadbbade7cb8dfbfbfe171504c8094308a2506fe477ec3939d/pystributed-0.1.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-01 16:03:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "yourusername",
"github_project": "pystributed",
"github_not_found": true,
"lcname": "pystributed"
}
JSON
