# aind-codeocean-utils
[](LICENSE)
[](https://github.com/semantic-release/semantic-release)
Library to contain useful utility methods to interface with Code Ocean.
## Installation
To use the package, you can install it from `pypi`:
```bash
pip install aind-codeocean-utils
```
To install the package from source, in the root directory, run
```bash
pip install -e .
```
To develop the code, run
```bash
pip install -e .[dev]
```
## Usage
The package includes helper functions to interact with Code Ocean:
### `CodeOceanJob`
This class enables one to run a job that:
1. Registers a new asset to Code Ocean from s3
2. Runs a capsule/pipeline on the newly registered asset (or an existing assey)
3. Captures the run results into a new asset
Steps 1 and 3 are optional, while step 2 (running the computation) is mandatory.
Here is a full example that registers a new ecephys asset, runs the spike sorting
capsule with some parameters, and registers the results:
```python
import os
from aind_codeocean_api.codeocean import CodeOceanClient
from aind_codeocean_utils.codeocean_job import (
CodeOceanJob, CodeOceanJobConfig
)
# Set up the CodeOceanClient from aind_codeocean_api
CO_TOKEN = os.environ["CO_TOKEN"]
CO_DOMAIN = os.environ["CO_DOMAIN"]
co_client = CodeOceanClient(domain=CO_DOMAIN, token=CO_TOKEN)
# Define Job Parameters
job_config_dict = dict(
register_config = dict(
asset_name="test_dataset_for_codeocean_job",
mount="ecephys_701305_2023-12-26_12-22-25",
bucket="aind-ephys-data",
prefix="ecephys_701305_2023-12-26_12-22-25",
tags=["codeocean_job_test", "ecephys", "701305", "raw"],
custom_metadata={
"modality": "extracellular electrophysiology",
"data level": "raw data",
},
viewable_to_everyone=True
),
run_capsule_config = dict(
data_assets=None, # when None, the newly registered asset will be used
capsule_id="a31e6c81-49a5-4f1c-b89c-2d47ae3e02b4",
run_parameters=["--debug", "--no-remove-out-channels"]
),
capture_result_config = dict(
process_name="sorted",
tags=["np-ultra"] # additional tags to the ones inherited from input
)
)
# instantiate config model
job_config = CodeOceanJobConfig(**job_config_dict)
# instantiate code ocean job
co_job = CodeOceanJob(co_client=co_client, job_config=job_config)
# run and wait for results
job_response = co_job.run_job()
```
This job will:
1. Register the `test_dataset_for_codeocean_job` asset from the specified s3 bucket and prefix
2. Run the capsule `a31e6c81-49a5-4f1c-b89c-2d47ae3e02b4` with the specified parameters
3. Register the result as `test_dataset_for_codeocean_job_sorter_{date-time}`
To run a computation on existing data assets, do not provide the `register_config` and
provide the `data_asset` field in the `run_capsule_config`.
To skip capturing the result, do not provide the `capture_result_config` option.
## Contributing
### Linters and testing
There are several libraries used to run linters, check documentation, and run tests.
- Please test your changes using the **coverage** library, which will run the tests and log a coverage report:
```bash
coverage run -m unittest discover && coverage report
```
- Use **interrogate** to check that modules, methods, etc. have been documented thoroughly:
```bash
interrogate .
```
- Use **flake8** to check that code is up to standards (no unused imports, etc.):
```bash
flake8 .
```
- Use **black** to automatically format the code into PEP standards:
```bash
black .
```
- Use **isort** to automatically sort import statements:
```bash
isort .
```
### Pull requests
For internal members, please create a branch. For external members, please fork the repository and open a pull request from the fork. We'll primarily use [Angular](https://github.com/angular/angular/blob/main/CONTRIBUTING.md#commit) style for commit messages. Roughly, they should follow the pattern:
```text
<type>(<scope>): <short summary>
```
where scope (optional) describes the packages affected by the code changes and type (mandatory) is one of:
- **build**: Changes that affect build tools or external dependencies (example scopes: pyproject.toml, setup.py)
- **ci**: Changes to our CI configuration files and scripts (examples: .github/workflows/ci.yml)
- **docs**: Documentation only changes
- **feat**: A new feature
- **fix**: A bugfix
- **perf**: A code change that improves performance
- **refactor**: A code change that neither fixes a bug nor adds a feature
- **test**: Adding missing tests or correcting existing tests
### Semantic Release
The table below, from [semantic release](https://github.com/semantic-release/semantic-release), shows which commit message gets you which release type when `semantic-release` runs (using the default configuration):
| Commit message | Release type |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- |
| `fix(pencil): stop graphite breaking when too much pressure applied` | ~~Patch~~ Fix Release, Default release |
| `feat(pencil): add 'graphiteWidth' option` | ~~Minor~~ Feature Release |
| `perf(pencil): remove graphiteWidth option`<br><br>`BREAKING CHANGE: The graphiteWidth option has been removed.`<br>`The default graphite width of 10mm is always used for performance reasons.` | ~~Major~~ Breaking Release <br /> (Note that the `BREAKING CHANGE: ` token must be in the footer of the commit) |
### Documentation
To generate the rst files source files for documentation, run
```bash
sphinx-apidoc -o doc_template/source/ src
```
Then to create the documentation HTML files, run
```bash
sphinx-build -b html doc_template/source/ doc_template/build/html
```
More info on sphinx installation can be found [here](https://www.sphinx-doc.org/en/master/usage/installation.html).
Raw data
{
"_id": null,
"home_page": null,
"name": "aind-codeocean-utils",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": null,
"author": "Allen Institute for Neural Dynamics",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/cc/25/bd666d0b60e5680ac2e60edd3c9cfdb2771694f61aa3de574d584e32055d/aind_codeocean_utils-0.2.0.tar.gz",
"platform": null,
"description": "# aind-codeocean-utils\n\n[](LICENSE)\n\n[](https://github.com/semantic-release/semantic-release)\n\n\n\n\nLibrary to contain useful utility methods to interface with Code Ocean.\n\n## Installation\n\nTo use the package, you can install it from `pypi`:\n```bash\npip install aind-codeocean-utils\n```\n\n\nTo install the package from source, in the root directory, run\n```bash\npip install -e .\n```\n\nTo develop the code, run\n```bash\npip install -e .[dev]\n```\n\n## Usage\n\nThe package includes helper functions to interact with Code Ocean:\n\n### `CodeOceanJob`\n\nThis class enables one to run a job that:\n\n1. Registers a new asset to Code Ocean from s3\n2. Runs a capsule/pipeline on the newly registered asset (or an existing assey)\n3. Captures the run results into a new asset\n\nSteps 1 and 3 are optional, while step 2 (running the computation) is mandatory.\n\nHere is a full example that registers a new ecephys asset, runs the spike sorting\ncapsule with some parameters, and registers the results:\n\n```python\nimport os\n\nfrom aind_codeocean_api.codeocean import CodeOceanClient\nfrom aind_codeocean_utils.codeocean_job import (\n CodeOceanJob, CodeOceanJobConfig\n)\n\n# Set up the CodeOceanClient from aind_codeocean_api\nCO_TOKEN = os.environ[\"CO_TOKEN\"]\nCO_DOMAIN = os.environ[\"CO_DOMAIN\"]\n\nco_client = CodeOceanClient(domain=CO_DOMAIN, token=CO_TOKEN)\n\n# Define Job Parameters\njob_config_dict = dict(\n register_config = dict(\n asset_name=\"test_dataset_for_codeocean_job\",\n mount=\"ecephys_701305_2023-12-26_12-22-25\",\n bucket=\"aind-ephys-data\",\n prefix=\"ecephys_701305_2023-12-26_12-22-25\",\n tags=[\"codeocean_job_test\", \"ecephys\", \"701305\", \"raw\"],\n custom_metadata={\n \"modality\": \"extracellular electrophysiology\",\n \"data level\": \"raw data\",\n },\n viewable_to_everyone=True\n ),\n run_capsule_config = dict(\n data_assets=None, # when None, the newly registered asset will be used\n capsule_id=\"a31e6c81-49a5-4f1c-b89c-2d47ae3e02b4\",\n run_parameters=[\"--debug\", \"--no-remove-out-channels\"]\n ),\n capture_result_config = dict(\n process_name=\"sorted\",\n tags=[\"np-ultra\"] # additional tags to the ones inherited from input\n )\n)\n\n# instantiate config model\njob_config = CodeOceanJobConfig(**job_config_dict)\n\n# instantiate code ocean job\nco_job = CodeOceanJob(co_client=co_client, job_config=job_config)\n\n# run and wait for results\njob_response = co_job.run_job()\n```\n\nThis job will:\n1. Register the `test_dataset_for_codeocean_job` asset from the specified s3 bucket and prefix\n2. Run the capsule `a31e6c81-49a5-4f1c-b89c-2d47ae3e02b4` with the specified parameters\n3. Register the result as `test_dataset_for_codeocean_job_sorter_{date-time}`\n\n\nTo run a computation on existing data assets, do not provide the `register_config` and\nprovide the `data_asset` field in the `run_capsule_config`.\n\nTo skip capturing the result, do not provide the `capture_result_config` option.\n\n\n## Contributing\n\n### Linters and testing\n\nThere are several libraries used to run linters, check documentation, and run tests.\n\n- Please test your changes using the **coverage** library, which will run the tests and log a coverage report:\n\n```bash\ncoverage run -m unittest discover && coverage report\n```\n\n- Use **interrogate** to check that modules, methods, etc. have been documented thoroughly:\n\n```bash\ninterrogate .\n```\n\n- Use **flake8** to check that code is up to standards (no unused imports, etc.):\n```bash\nflake8 .\n```\n\n- Use **black** to automatically format the code into PEP standards:\n```bash\nblack .\n```\n\n- Use **isort** to automatically sort import statements:\n```bash\nisort .\n```\n\n### Pull requests\n\nFor internal members, please create a branch. For external members, please fork the repository and open a pull request from the fork. We'll primarily use [Angular](https://github.com/angular/angular/blob/main/CONTRIBUTING.md#commit) style for commit messages. Roughly, they should follow the pattern:\n```text\n<type>(<scope>): <short summary>\n```\n\nwhere scope (optional) describes the packages affected by the code changes and type (mandatory) is one of:\n\n- **build**: Changes that affect build tools or external dependencies (example scopes: pyproject.toml, setup.py)\n- **ci**: Changes to our CI configuration files and scripts (examples: .github/workflows/ci.yml)\n- **docs**: Documentation only changes\n- **feat**: A new feature\n- **fix**: A bugfix\n- **perf**: A code change that improves performance\n- **refactor**: A code change that neither fixes a bug nor adds a feature\n- **test**: Adding missing tests or correcting existing tests\n\n### Semantic Release\n\nThe table below, from [semantic release](https://github.com/semantic-release/semantic-release), shows which commit message gets you which release type when `semantic-release` runs (using the default configuration):\n\n| Commit message | Release type |\n| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- |\n| `fix(pencil): stop graphite breaking when too much pressure applied` | ~~Patch~~ Fix Release, Default release |\n| `feat(pencil): add 'graphiteWidth' option` | ~~Minor~~ Feature Release |\n| `perf(pencil): remove graphiteWidth option`<br><br>`BREAKING CHANGE: The graphiteWidth option has been removed.`<br>`The default graphite width of 10mm is always used for performance reasons.` | ~~Major~~ Breaking Release <br /> (Note that the `BREAKING CHANGE: ` token must be in the footer of the commit) |\n\n### Documentation\nTo generate the rst files source files for documentation, run\n```bash\nsphinx-apidoc -o doc_template/source/ src \n```\nThen to create the documentation HTML files, run\n```bash\nsphinx-build -b html doc_template/source/ doc_template/build/html\n```\nMore info on sphinx installation can be found [here](https://www.sphinx-doc.org/en/master/usage/installation.html).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Generated from aind-library-template",
"version": "0.2.0",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "dad5bee75638fc7305c89c11d172f0650340099f9e2c03d52f057571437f1535",
"md5": "6aa2c82c34c2bb2855f6fd8228c1039a",
"sha256": "ae9c8a40c99f3530b20ac3deac185dfd50a03bdc4d40263f9a9fa9f0705f0221"
},
"downloads": -1,
"filename": "aind_codeocean_utils-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6aa2c82c34c2bb2855f6fd8228c1039a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 13789,
"upload_time": "2024-08-26T14:33:02",
"upload_time_iso_8601": "2024-08-26T14:33:02.494384Z",
"url": "https://files.pythonhosted.org/packages/da/d5/bee75638fc7305c89c11d172f0650340099f9e2c03d52f057571437f1535/aind_codeocean_utils-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "cc25bd666d0b60e5680ac2e60edd3c9cfdb2771694f61aa3de574d584e32055d",
"md5": "74d420ab21b910aae79aa80a252ae6b1",
"sha256": "d1d1c72dcbcc75d1448d0bc5d6e84a5d35d7e868009504c16201e2aa6f5707ba"
},
"downloads": -1,
"filename": "aind_codeocean_utils-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "74d420ab21b910aae79aa80a252ae6b1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 50510,
"upload_time": "2024-08-26T14:33:03",
"upload_time_iso_8601": "2024-08-26T14:33:03.885839Z",
"url": "https://files.pythonhosted.org/packages/cc/25/bd666d0b60e5680ac2e60edd3c9cfdb2771694f61aa3de574d584e32055d/aind_codeocean_utils-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-26 14:33:03",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "aind-codeocean-utils"
}
JSON