# Cumulus Message Adapter
[](https://circleci.com/gh/nasa/cumulus-message-adapter)
`cumulus-message-adapter` is a command-line interface for preparing and outputting Cumulus Messages for Cumulus Tasks. `cumulus-message-adapter` helps Cumulus developers integrate a task into a Cumulus Workflow.
Read more about how the `cumulus-message-adapter` works in the [CONTRACT.md](./CONTRACT.md).
## Releases
### Release Versions
Please note the following convention for release versions:
X.Y.Z: where:
* X is an organizational release that signifies the completion of a core set of functionality
* Y is a major version release that may include incompatible API changes and/or other breaking changes
* Z is a minor version that includes bugfixes and backwards compatible improvements
### Continuous Integration
[CircleCI](https://circleci.com/gh/nasa/cumulus-message-adapter) manages releases and release assets.
Whenever CircleCI passes on the master branch of cumulus-message-adapter and `message_adapter/version.py` has been updated with a version that doesn't match an existing tag, CircleCI will:
* Create a new tag with `tag_name` of the string in `message_adapter/version.py`
* Create a new release using the new tag, with a name equal to `tag_name` (equal to version).
* Build a `cumulus-message-adapter.zip` file and attach it as a release asset to the newly created release. The zip file is created using the [`Makefile`](./Makefile) in the root of this repository.
These steps are fully detailed in the [`.circleci/config.yml`](./.circleci/config.yml) file.
## Development
### Dependency Installation
```shell
pip install -r requirements-dev.txt
pip install -r requirements.txt
```
### Running Tests
Running tests requires [localstack](https://github.com/localstack/localstack).
Tests only require localstack running S3, which can be initiated with the following command:
```shell
EAGER_SERVICE_LOADING=1 SERVICES=s3 localstack start
```
And then you can check tests pass with the following nosetests command:
```shell
CUMULUS_ENV=testing nose2 -v
```
### Linting
```shell
pylint message_adapter
```
### Contributing
If changes are made to the codebase, you can create the cumulus-message-adapter zip archive for testing libraries that require it:
```shell
make clean
make cumulus-message-adapter.zip
```
Then you can run some integration tests:
```shell
./examples/example-node-message-adapter-lib.js
```
Before any changes are finalized and released, they should be tested by packaging the cumulus-message-adapter zip archive and testing it in a lambda environment, as that is where it will be utilized.
#### Packaging
Packaging the zip file is probably best done in an environment that closely matches the lambda environment in which it will be run and contains the current Python version, so we are using an AWS Python Lambda image. Certain packages need to be installed, and using a virtual environment is important due to Python pathing.
```shell
docker run -v ~/projects/cumulus-message-adapter/:/cma/ -v ~/tmp/:/tmp/ -v ~/amazon/:/home/amazon/ -it --entrypoint /bin/bash amazon/aws-lambda-python:3.10
yum install -y make binutils zip
cd /cma
pip install --user virtualenv
~/.local/bin/virtualenv ~/venv310
. ~/venv310/bin/activate
pip install .
make clean
make cumulus-message-adapter.zip
```
#### Testing the package in a Lambda Environment
Once the package is created, it should be tested in a Lambda environment. Before doing so, it may be helpful to run the package in the container it was packaged in, immediately after the above commands to see if any errors occur, which will indicate an issue in creating the package `./dist/cma stream`.
If no errors occur immediately, you can optionally test the zip in an AWS Lambda NodeJS image, as that is the target environment. Running in an image may allow for quicker testing and development, but testing in AWS should still be the final test.
```shell
docker run -v ~/projects/cumulus-message-adapter:/zipfile --entrypoint /
bin/bash -it amazon/aws-lambda-nodejs:16
cd /zipfile
cp -r dist /opt/
cd /opt/dist
./cma stream
```
Testing the package in AWS Lambda requires uploading the zip as a layer and then running a Cumulus step function that utilizes that layer. The following instructions are for Cumulus Core team members that have access to a layer specifically set up for this purpose.
* In the AWS console, go to Lambda > Layers > CMA_Test
* Create a new version by uploading the cumulus-message-adapater zip file packaged earlier
* In your /cumulus-tf/terraform.tfvars, replace the `cumulus_message_adapter_lambda_layer_version_arn` value with the newly created Version ARN
* Apply the change with `terraform apply`
* Find any recent successfully run Step Function, and run a New Execution. The 'Functions using this version' tab of the CMA_TEST layer should provide some options.
### Troubleshooting
* Error: "DistutilsOptionError: must supply either home or prefix/exec-prefix — not both" when running `make cumulus-message-adapter.zip`
* [Solution](https://stackoverflow.com/a/24357384)
Raw data
{
"_id": null,
"home_page": "https://github.com/nasa/cumulus-message-adapter",
"name": "cumulus-message-adapter",
"maintainer": "",
"docs_url": null,
"requires_python": "~=3.10",
"maintainer_email": "",
"keywords": "nasa cumulus message adapter",
"author": "Cumulus Authors",
"author_email": "info@developmentseed.org",
"download_url": "https://files.pythonhosted.org/packages/ac/2c/b108d976fa8a43d88e2fc78d0076708be5144d56456b1dc73feb33080ed7/cumulus-message-adapter-2.0.3.post1.tar.gz",
"platform": null,
"description": "# Cumulus Message Adapter\n\n[](https://circleci.com/gh/nasa/cumulus-message-adapter)\n\n`cumulus-message-adapter` is a command-line interface for preparing and outputting Cumulus Messages for Cumulus Tasks. `cumulus-message-adapter` helps Cumulus developers integrate a task into a Cumulus Workflow.\n\nRead more about how the `cumulus-message-adapter` works in the [CONTRACT.md](./CONTRACT.md).\n\n## Releases\n\n### Release Versions\n\nPlease note the following convention for release versions:\n\nX.Y.Z: where:\n\n* X is an organizational release that signifies the completion of a core set of functionality\n* Y is a major version release that may include incompatible API changes and/or other breaking changes\n* Z is a minor version that includes bugfixes and backwards compatible improvements\n\n### Continuous Integration\n\n[CircleCI](https://circleci.com/gh/nasa/cumulus-message-adapter) manages releases and release assets.\n\nWhenever CircleCI passes on the master branch of cumulus-message-adapter and `message_adapter/version.py` has been updated with a version that doesn't match an existing tag, CircleCI will:\n\n* Create a new tag with `tag_name` of the string in `message_adapter/version.py`\n* Create a new release using the new tag, with a name equal to `tag_name` (equal to version).\n* Build a `cumulus-message-adapter.zip` file and attach it as a release asset to the newly created release. The zip file is created using the [`Makefile`](./Makefile) in the root of this repository.\n\nThese steps are fully detailed in the [`.circleci/config.yml`](./.circleci/config.yml) file.\n\n## Development\n\n### Dependency Installation\n\n```shell\npip install -r requirements-dev.txt\npip install -r requirements.txt\n```\n\n### Running Tests\n\nRunning tests requires [localstack](https://github.com/localstack/localstack).\n\nTests only require localstack running S3, which can be initiated with the following command:\n\n```shell\nEAGER_SERVICE_LOADING=1 SERVICES=s3 localstack start\n```\n\nAnd then you can check tests pass with the following nosetests command:\n\n```shell\nCUMULUS_ENV=testing nose2 -v\n```\n\n### Linting\n\n```shell\npylint message_adapter\n```\n\n### Contributing\n\nIf changes are made to the codebase, you can create the cumulus-message-adapter zip archive for testing libraries that require it:\n\n```shell\nmake clean\nmake cumulus-message-adapter.zip\n```\n\nThen you can run some integration tests:\n\n```shell\n./examples/example-node-message-adapter-lib.js\n```\n\nBefore any changes are finalized and released, they should be tested by packaging the cumulus-message-adapter zip archive and testing it in a lambda environment, as that is where it will be utilized.\n\n#### Packaging\n\nPackaging the zip file is probably best done in an environment that closely matches the lambda environment in which it will be run and contains the current Python version, so we are using an AWS Python Lambda image. Certain packages need to be installed, and using a virtual environment is important due to Python pathing.\n\n```shell\ndocker run -v ~/projects/cumulus-message-adapter/:/cma/ -v ~/tmp/:/tmp/ -v ~/amazon/:/home/amazon/ -it --entrypoint /bin/bash amazon/aws-lambda-python:3.10\nyum install -y make binutils zip\ncd /cma\npip install --user virtualenv\n~/.local/bin/virtualenv ~/venv310\n. ~/venv310/bin/activate\npip install .\nmake clean\nmake cumulus-message-adapter.zip\n```\n\n#### Testing the package in a Lambda Environment\n\nOnce the package is created, it should be tested in a Lambda environment. Before doing so, it may be helpful to run the package in the container it was packaged in, immediately after the above commands to see if any errors occur, which will indicate an issue in creating the package `./dist/cma stream`.\n\nIf no errors occur immediately, you can optionally test the zip in an AWS Lambda NodeJS image, as that is the target environment. Running in an image may allow for quicker testing and development, but testing in AWS should still be the final test.\n\n```shell\ndocker run -v ~/projects/cumulus-message-adapter:/zipfile --entrypoint /\nbin/bash -it amazon/aws-lambda-nodejs:16\ncd /zipfile\ncp -r dist /opt/\ncd /opt/dist\n./cma stream\n```\n\nTesting the package in AWS Lambda requires uploading the zip as a layer and then running a Cumulus step function that utilizes that layer. The following instructions are for Cumulus Core team members that have access to a layer specifically set up for this purpose.\n\n* In the AWS console, go to Lambda > Layers > CMA_Test\n* Create a new version by uploading the cumulus-message-adapater zip file packaged earlier\n* In your /cumulus-tf/terraform.tfvars, replace the `cumulus_message_adapter_lambda_layer_version_arn` value with the newly created Version ARN\n* Apply the change with `terraform apply`\n* Find any recent successfully run Step Function, and run a New Execution. The 'Functions using this version' tab of the CMA_TEST layer should provide some options.\n\n### Troubleshooting\n\n* Error: \"DistutilsOptionError: must supply either home or prefix/exec-prefix \u2014 not both\" when running `make cumulus-message-adapter.zip`\n * [Solution](https://stackoverflow.com/a/24357384)\n",
"bugtrack_url": null,
"license": "",
"summary": "A command-line interface for preparing and outputting Cumulus Messages for Cumulus Tasks",
"version": "2.0.3.post1",
"project_urls": {
"Homepage": "https://github.com/nasa/cumulus-message-adapter"
},
"split_keywords": [
"nasa",
"cumulus",
"message",
"adapter"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ac2cb108d976fa8a43d88e2fc78d0076708be5144d56456b1dc73feb33080ed7",
"md5": "118f7e3be8b33cc7ac9415dbea38696c",
"sha256": "3f02a5d1fe0a299a552213672ff1e6d3ca72330aa158f9bf4669247caab4f741"
},
"downloads": -1,
"filename": "cumulus-message-adapter-2.0.3.post1.tar.gz",
"has_sig": false,
"md5_digest": "118f7e3be8b33cc7ac9415dbea38696c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "~=3.10",
"size": 19355,
"upload_time": "2023-10-12T19:05:23",
"upload_time_iso_8601": "2023-10-12T19:05:23.860771Z",
"url": "https://files.pythonhosted.org/packages/ac/2c/b108d976fa8a43d88e2fc78d0076708be5144d56456b1dc73feb33080ed7/cumulus-message-adapter-2.0.3.post1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-12 19:05:23",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "nasa",
"github_project": "cumulus-message-adapter",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"circle": true,
"requirements": [],
"lcname": "cumulus-message-adapter"
}
JSON
