# AML Pipeline Local Run Guide
This repository contains sdk code to run aml pipeline locally.
So far there are 2 job running modes supported in pipeline local run:
* Native: this mode means run the job in native process
* Container: this mode the local pipeline executor will help building the container based on the environment defined in pipeline job component.
**limitations:**
* Only support CommandJob (SweepJob and DistributedJob are not supported)
* Pure local run with local metrics/UI support (need to run local server container if want to see ui)
## project structure
- piprunengine: main pkg for pipeline local executor
- tests: unit test to run all sample pipelines (defined in notebookxx)
- notebookxx: unit test pipeline definition and related resources.
## prerequisites
* Docker
* azure ml devplat-v2 sdk
* python version > 3.7
## How to run it
### 1.1 clone this repo to local
### 1.2 create a new python env with conda
```shell
conda create -n <local_run_env> python=3.9
```
### 1.3 install the local run sdk
find the latest wheel in release folder and install it in the new created env
```shell
pip install pipelinelocalrun==0.1.9
```
### 1.4 install jupyter if want to run in notebook
```shell
pip install jupyter
```
### 1.5 start local web server if want to try local ui & mlflow (optional)
```shell
docker run -e AML_LOCAL_RUN_DB_PATH="/metadata/localrun.db" -e LOCALUI_START=true -p 127.0.0.1:4999:80/tcp -p 127.0.0.1:8080:8080/tcp -p 127.0.0.1:3001:3001/tcp -p 127.0.0.1:5000:5000/tcp --mount type=bind,source='<current-user-home-path>\.azureml\piprun',target=/metadata <image_name:tag>
```
`<current-user-home-path>`:
* for windows: `C:\Users\<username>`
* for linux: `/home/<username>`
`<image_name:tag>` is the image built in step 2.4
### 1.6 how to use in code/notebook
* import the local run func from the install pkg
```python
from piprunengine import run
```
* following the normal steps in the notebook to build your pipeline(you can ignore all steps which need interaction with aml workspace)
* start pipeline local run
```python
output_root_dir = "./local-run-output/notebook_1a_native"
# set pipeline name
pipeline.name="test-pipeline"
run(output_root_dir=output_root_dir, job=pipeline, experiment_name="test")
```
`note`: by default it runs in CONTAINER mode and it will build the curated container locally and run in container mode.
### 1.7 try with notebook demo
* [notebook1a/pipeline_with_components_from_yaml.ipynb](./notebook1a/pipeline_with_components_from_yaml.ipynb)
* [notebook1b/pipeline_with_python_function_components.ipynb](./notebook1b/pipeline_with_python_function_components.ipynb)
* [notebook2c/nyc_taxi_data_regression.ipynb](./notebook2c/nyc_taxi_data_regression.ipynb)
* [notebook1d/pipeline_with_non_python_components.ipynb](./notebook1d/pipeline_with_non_python_components.ipynb)
## Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
## Trademarks
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must follow
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.
Raw data
{
"_id": null,
"home_page": "https://github.com/Azure/aml_pipeline_local_run",
"name": "pipelinelocalrun",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "Xiaopeng Wang",
"author_email": "xiaopwan@microsoft.com",
"download_url": "https://files.pythonhosted.org/packages/65/74/0f8105b4512bea77d6e1f0d967be890b23aa6aebc32152ac0a5a78d46fce/pipelinelocalrun-0.2.0.tar.gz",
"platform": null,
"description": "# AML Pipeline Local Run Guide\n\nThis repository contains sdk code to run aml pipeline locally.\nSo far there are 2 job running modes supported in pipeline local run:\n\n* Native: this mode means run the job in native process\n* Container: this mode the local pipeline executor will help building the container based on the environment defined in pipeline job component.\n\n**limitations:**\n* Only support CommandJob (SweepJob and DistributedJob are not supported)\n* Pure local run with local metrics/UI support (need to run local server container if want to see ui)\n\n## project structure\n\n- piprunengine: main pkg for pipeline local executor\n- tests: unit test to run all sample pipelines (defined in notebookxx)\n- notebookxx: unit test pipeline definition and related resources. \n\n## prerequisites\n\n* Docker\n* azure ml devplat-v2 sdk\n* python version > 3.7\n\n\n## How to run it\n\n### 1.1 clone this repo to local\n\n### 1.2 create a new python env with conda\n\n```shell\nconda create -n <local_run_env> python=3.9\n```\n\n### 1.3 install the local run sdk\n\nfind the latest wheel in release folder and install it in the new created env\n\n```shell\npip install pipelinelocalrun==0.1.9\n```\n\n### 1.4 install jupyter if want to run in notebook\n\n```shell\npip install jupyter\n```\n\n### 1.5 start local web server if want to try local ui & mlflow (optional)\n\n```shell\ndocker run -e AML_LOCAL_RUN_DB_PATH=\"/metadata/localrun.db\" -e LOCALUI_START=true -p 127.0.0.1:4999:80/tcp -p 127.0.0.1:8080:8080/tcp -p 127.0.0.1:3001:3001/tcp -p 127.0.0.1:5000:5000/tcp --mount type=bind,source='<current-user-home-path>\\.azureml\\piprun',target=/metadata <image_name:tag>\n```\n\n `<current-user-home-path>`: \n * for windows: `C:\\Users\\<username>`\n * for linux: `/home/<username>`\n\n`<image_name:tag>` is the image built in step 2.4\n\n### 1.6 how to use in code/notebook\n\n* import the local run func from the install pkg\n \n ```python\n from piprunengine import run\n ```\n\n* following the normal steps in the notebook to build your pipeline(you can ignore all steps which need interaction with aml workspace)\n\n* start pipeline local run\n \n ```python\n output_root_dir = \"./local-run-output/notebook_1a_native\"\n # set pipeline name\n pipeline.name=\"test-pipeline\"\n run(output_root_dir=output_root_dir, job=pipeline, experiment_name=\"test\")\n ```\n\n `note`: by default it runs in CONTAINER mode and it will build the curated container locally and run in container mode.\n\n### 1.7 try with notebook demo\n* [notebook1a/pipeline_with_components_from_yaml.ipynb](./notebook1a/pipeline_with_components_from_yaml.ipynb)\n* [notebook1b/pipeline_with_python_function_components.ipynb](./notebook1b/pipeline_with_python_function_components.ipynb)\n* [notebook2c/nyc_taxi_data_regression.ipynb](./notebook2c/nyc_taxi_data_regression.ipynb)\n* [notebook1d/pipeline_with_non_python_components.ipynb](./notebook1d/pipeline_with_non_python_components.ipynb)\n## Contributing\n\nThis project welcomes contributions and suggestions. Most contributions require you to agree to a\nContributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\nthe rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.\n\nWhen you submit a pull request, a CLA bot will automatically determine whether you need to provide\na CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions\nprovided by the bot. You will only need to do this once across all repos using our CLA.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\ncontact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\n\n## Trademarks\n\nThis project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft \ntrademarks or logos is subject to and must follow \n[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).\nUse of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.\nAny use of third-party trademarks or logos are subject to those third-party's policies.\n",
"bugtrack_url": null,
"license": "",
"summary": "Package to support pipeline local run",
"version": "0.2.0",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3a1228cf6ba3c791c750b7c0a4e024667719473505341bd8dd7096bda6971e3a",
"md5": "3d240fe3b772202962192d374712d7f3",
"sha256": "005298e208e21732a04d98d6a9a8f04b82cf730a52e2734f63cc2db3ccdee8f1"
},
"downloads": -1,
"filename": "pipelinelocalrun-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3d240fe3b772202962192d374712d7f3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 44897,
"upload_time": "2023-02-01T09:49:11",
"upload_time_iso_8601": "2023-02-01T09:49:11.930247Z",
"url": "https://files.pythonhosted.org/packages/3a/12/28cf6ba3c791c750b7c0a4e024667719473505341bd8dd7096bda6971e3a/pipelinelocalrun-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "65740f8105b4512bea77d6e1f0d967be890b23aa6aebc32152ac0a5a78d46fce",
"md5": "ca396cd0cccf41f8a379b85e6091dffe",
"sha256": "8925eac2801a9433dd01ff77918805895e90656cc5ab2d5770a916d9b65ff966"
},
"downloads": -1,
"filename": "pipelinelocalrun-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "ca396cd0cccf41f8a379b85e6091dffe",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 34104,
"upload_time": "2023-02-01T09:49:13",
"upload_time_iso_8601": "2023-02-01T09:49:13.769395Z",
"url": "https://files.pythonhosted.org/packages/65/74/0f8105b4512bea77d6e1f0d967be890b23aa6aebc32152ac0a5a78d46fce/pipelinelocalrun-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-02-01 09:49:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "Azure",
"github_project": "aml_pipeline_local_run",
"lcname": "pipelinelocalrun"
}
JSON
