# aind-codeocean-pipeline-monitor
[](LICENSE)
[](https://github.com/semantic-release/semantic-release)
Package for starting a pipeline, waiting for it to finish, and optionally capturing the results as a data asset.
## Installation
The repo can be install from PyPI. To pip install all of the necessary dependencies to run the pipeline monitor, run:
```bash
pip install aind-codeocean-pipeline-monitor[full]
```
To install only the minimum dependencies required for model validation, run:
```bash
pip install aind-codeocean-pipeline-monitor
```
To install the package for development, clone this repo and run
```bash
pip install -e .[dev]
```
## Usage
- Define job using PipelineMonitorJobSettings class.
- Define a CodeOcean client.
- Construct a PipelineMonitorJob with these settings.
- Run the job with the run_job method.
```python
import os
from codeocean import CodeOcean
from codeocean.computation import DataAssetsRunParam, RunParams
from urllib3.util import Retry
from aind_codeocean_pipeline_monitor.job import PipelineMonitorJob
from aind_codeocean_pipeline_monitor.models import (
CaptureSettings,
PipelineMonitorSettings,
)
domain = os.getenv("CODEOCEAN_DOMAIN")
token = os.getenv("CODEOCEAN_TOKEN")
# Recommend adding retry strategy to requests session
retry = Retry(
total=5,
backoff_factor=1,
status_forcelist=[429, 500, 502, 503, 504],
allowed_methods=["GET", "POST"],
)
client = CodeOcean(domain=domain, token=token, retries=retry)
# Please consult Code Ocean docs for info about RunParams and DataAssetParams
settings = PipelineMonitorSettings(
run_params=RunParams(
capsule_id="<your capsule id>",
data_assets=[
DataAssetsRunParam(
id="<your input data asset id>",
mount="<your input data mount>",
)
],
),
capture_settings=CaptureSettings(
tags=["derived"]
), # 'tags' is the only required field
)
job = PipelineMonitorJob(job_settings=settings, client=client)
job.run_job()
```
## Contributing
### Linters and testing
There are several libraries used to run linters, check documentation, and run tests.
- Please test your changes using the **coverage** library, which will run the tests and log a coverage report:
```bash
coverage run -m unittest discover && coverage report
```
- Use **interrogate** to check that modules, methods, etc. have been documented thoroughly:
```bash
interrogate .
```
- Use **flake8** to check that code is up to standards (no unused imports, etc.):
```bash
flake8 .
```
- Use **black** to automatically format the code into PEP standards:
```bash
black .
```
- Use **isort** to automatically sort import statements:
```bash
isort .
```
### Pull requests
For internal members, please create a branch. For external members, please fork the repository and open a pull request from the fork. We'll primarily use [Angular](https://github.com/angular/angular/blob/main/CONTRIBUTING.md#commit) style for commit messages. Roughly, they should follow the pattern:
```text
<type>(<scope>): <short summary>
```
where scope (optional) describes the packages affected by the code changes and type (mandatory) is one of:
- **build**: Changes that affect build tools or external dependencies (example scopes: pyproject.toml, setup.py)
- **ci**: Changes to our CI configuration files and scripts (examples: .github/workflows/ci.yml)
- **docs**: Documentation only changes
- **feat**: A new feature
- **fix**: A bugfix
- **perf**: A code change that improves performance
- **refactor**: A code change that neither fixes a bug nor adds a feature
- **test**: Adding missing tests or correcting existing tests
### Semantic Release
The table below, from [semantic release](https://github.com/semantic-release/semantic-release), shows which commit message gets you which release type when `semantic-release` runs (using the default configuration):
| Commit message | Release type |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- |
| `fix(pencil): stop graphite breaking when too much pressure applied` | ~~Patch~~ Fix Release, Default release |
| `feat(pencil): add 'graphiteWidth' option` | ~~Minor~~ Feature Release |
| `perf(pencil): remove graphiteWidth option`<br><br>`BREAKING CHANGE: The graphiteWidth option has been removed.`<br>`The default graphite width of 10mm is always used for performance reasons.` | ~~Major~~ Breaking Release <br /> (Note that the `BREAKING CHANGE: ` token must be in the footer of the commit) |
### Documentation
To generate the rst files source files for documentation, run
```bash
sphinx-apidoc -o docs/source/ src
```
Then to create the documentation HTML files, run
```bash
sphinx-build -b html docs/source/ docs/build/html
```
More info on sphinx installation can be found [here](https://www.sphinx-doc.org/en/master/usage/installation.html).
Raw data
{
"_id": null,
"home_page": null,
"name": "aind-codeocean-pipeline-monitor",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": "Allen Institute for Neural Dynamics",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/ab/35/33bcb6deb48d14f8d2ae3bfdf642e7cfb6b608985c5e8edeee329d05b916/aind_codeocean_pipeline_monitor-0.6.3.tar.gz",
"platform": null,
"description": "# aind-codeocean-pipeline-monitor\n\n[](LICENSE)\n\n[](https://github.com/semantic-release/semantic-release)\n\n\n\n\nPackage for starting a pipeline, waiting for it to finish, and optionally capturing the results as a data asset.\n\n## Installation\nThe repo can be install from PyPI. To pip install all of the necessary dependencies to run the pipeline monitor, run:\n```bash\npip install aind-codeocean-pipeline-monitor[full]\n```\n\nTo install only the minimum dependencies required for model validation, run:\n```bash\npip install aind-codeocean-pipeline-monitor\n```\n\nTo install the package for development, clone this repo and run\n```bash\npip install -e .[dev]\n```\n\n## Usage\n- Define job using PipelineMonitorJobSettings class.\n- Define a CodeOcean client.\n- Construct a PipelineMonitorJob with these settings.\n- Run the job with the run_job method.\n\n```python\nimport os\n\nfrom codeocean import CodeOcean\nfrom codeocean.computation import DataAssetsRunParam, RunParams\nfrom urllib3.util import Retry\n\nfrom aind_codeocean_pipeline_monitor.job import PipelineMonitorJob\nfrom aind_codeocean_pipeline_monitor.models import (\n CaptureSettings,\n PipelineMonitorSettings,\n)\n\ndomain = os.getenv(\"CODEOCEAN_DOMAIN\")\ntoken = os.getenv(\"CODEOCEAN_TOKEN\")\n# Recommend adding retry strategy to requests session\nretry = Retry(\n total=5,\n backoff_factor=1,\n status_forcelist=[429, 500, 502, 503, 504],\n allowed_methods=[\"GET\", \"POST\"],\n)\nclient = CodeOcean(domain=domain, token=token, retries=retry)\n\n# Please consult Code Ocean docs for info about RunParams and DataAssetParams\nsettings = PipelineMonitorSettings(\n run_params=RunParams(\n capsule_id=\"<your capsule id>\",\n data_assets=[\n DataAssetsRunParam(\n id=\"<your input data asset id>\",\n mount=\"<your input data mount>\",\n )\n ],\n ),\n capture_settings=CaptureSettings(\n tags=[\"derived\"]\n ), # 'tags' is the only required field\n)\n\njob = PipelineMonitorJob(job_settings=settings, client=client)\njob.run_job()\n```\n\n## Contributing\n\n### Linters and testing\n\nThere are several libraries used to run linters, check documentation, and run tests.\n\n- Please test your changes using the **coverage** library, which will run the tests and log a coverage report:\n\n```bash\ncoverage run -m unittest discover && coverage report\n```\n\n- Use **interrogate** to check that modules, methods, etc. have been documented thoroughly:\n\n```bash\ninterrogate .\n```\n\n- Use **flake8** to check that code is up to standards (no unused imports, etc.):\n```bash\nflake8 .\n```\n\n- Use **black** to automatically format the code into PEP standards:\n```bash\nblack .\n```\n\n- Use **isort** to automatically sort import statements:\n```bash\nisort .\n```\n\n### Pull requests\n\nFor internal members, please create a branch. For external members, please fork the repository and open a pull request from the fork. We'll primarily use [Angular](https://github.com/angular/angular/blob/main/CONTRIBUTING.md#commit) style for commit messages. Roughly, they should follow the pattern:\n```text\n<type>(<scope>): <short summary>\n```\n\nwhere scope (optional) describes the packages affected by the code changes and type (mandatory) is one of:\n\n- **build**: Changes that affect build tools or external dependencies (example scopes: pyproject.toml, setup.py)\n- **ci**: Changes to our CI configuration files and scripts (examples: .github/workflows/ci.yml)\n- **docs**: Documentation only changes\n- **feat**: A new feature\n- **fix**: A bugfix\n- **perf**: A code change that improves performance\n- **refactor**: A code change that neither fixes a bug nor adds a feature\n- **test**: Adding missing tests or correcting existing tests\n\n### Semantic Release\n\nThe table below, from [semantic release](https://github.com/semantic-release/semantic-release), shows which commit message gets you which release type when `semantic-release` runs (using the default configuration):\n\n| Commit message | Release type |\n| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- |\n| `fix(pencil): stop graphite breaking when too much pressure applied` | ~~Patch~~ Fix Release, Default release |\n| `feat(pencil): add 'graphiteWidth' option` | ~~Minor~~ Feature Release |\n| `perf(pencil): remove graphiteWidth option`<br><br>`BREAKING CHANGE: The graphiteWidth option has been removed.`<br>`The default graphite width of 10mm is always used for performance reasons.` | ~~Major~~ Breaking Release <br /> (Note that the `BREAKING CHANGE: ` token must be in the footer of the commit) |\n\n### Documentation\nTo generate the rst files source files for documentation, run\n```bash\nsphinx-apidoc -o docs/source/ src\n```\nThen to create the documentation HTML files, run\n```bash\nsphinx-build -b html docs/source/ docs/build/html\n```\nMore info on sphinx installation can be found [here](https://www.sphinx-doc.org/en/master/usage/installation.html).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Package to define and run a Code Ocean Pipeline Monitor Job",
"version": "0.6.3",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "eef1119e16d940041495ecef1bddebaee978a954401bec773217f02a5816bf5d",
"md5": "6b93706ea7beb6abcd82ecc0f60f5ff3",
"sha256": "8e0c6d7c573c52b165c8d918cc0c786405be46a9d8e565635d45976f31f322fc"
},
"downloads": -1,
"filename": "aind_codeocean_pipeline_monitor-0.6.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6b93706ea7beb6abcd82ecc0f60f5ff3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 11415,
"upload_time": "2025-03-01T20:01:21",
"upload_time_iso_8601": "2025-03-01T20:01:21.965242Z",
"url": "https://files.pythonhosted.org/packages/ee/f1/119e16d940041495ecef1bddebaee978a954401bec773217f02a5816bf5d/aind_codeocean_pipeline_monitor-0.6.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "ab3533bcb6deb48d14f8d2ae3bfdf642e7cfb6b608985c5e8edeee329d05b916",
"md5": "a64c0cd839ccc99c4bdd844e4f87590f",
"sha256": "2d114e85ce1030d76fbf8c1b878366b2818b5f60f533778a4056240e25ac7129"
},
"downloads": -1,
"filename": "aind_codeocean_pipeline_monitor-0.6.3.tar.gz",
"has_sig": false,
"md5_digest": "a64c0cd839ccc99c4bdd844e4f87590f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 47363,
"upload_time": "2025-03-01T20:01:23",
"upload_time_iso_8601": "2025-03-01T20:01:23.383051Z",
"url": "https://files.pythonhosted.org/packages/ab/35/33bcb6deb48d14f8d2ae3bfdf642e7cfb6b608985c5e8edeee329d05b916/aind_codeocean_pipeline_monitor-0.6.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-03-01 20:01:23",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "aind-codeocean-pipeline-monitor"
}
JSON