| Name | osml-data-intake JSON |
| Version |
1.0.0
JSON |
| download |
| home_page | None |
| Summary | Package to run the osml image intake component service. |
| upload_time | 2024-06-06 19:06:20 |
| maintainer | None |
| docs_url | None |
| author | Amazon Web Services |
| requires_python | >=3.9 |
| license | MIT No Attribution Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
|
# OSML Data Intake
## Overview
This application facilitates the processing, conversion, and management of satellite imagery metadata as part of the OversightML (OSML) framework and can be deployed as part of the OSML [guidance package](https://github.com/aws-solutions-library-samples/guidance-for-processing-overhead-imagery-on-aws). It leverages the GDAL library and integrates with Amazon S3 for seamless storage and sharing to provide imagery metadata to other service components. Below is an overview of the main features:
### Image Metadata Processing
Extracts and processes metadata from satellite imagery files, such as image dimensions and geographical coordinates. Generates auxiliary .aux and .ovr files for optimized image previews.
### SpatioTemporal Asset Catalogs (STAC) Item Generation
Incorporates converted coordinates into STAC-compliant data formats for indexing and sharing geospatial information. Automatically handles data formatting and polygon creation.
### S3 Output
Uploads auxiliary files and metadata to Amazon S3 using the boto3 SDK. Logs success and error messages for uploads to aid in debugging and auditing.
### Table of Contents
* [Getting Started](#getting-started)
* [Prerequisites](#prerequisites)
* [Installation Guide](#installation-guide)
* [Documentation](#documentation)
* [Testing Your Setup](#testing-your-setup)
* [Support & Feedback](#support--feedback)
* [Security](#security)
* [License](#license)
## Getting Started
### Prerequisites
First, ensure you have installed the following tools locally
1. [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
2. [docker](https://nodejs.org/en)
3. [tox](https://tox.wiki/en/latest/installation.html)
### Installation Guide
1. Clone `osml-data-intake` package into your desktop
```sh
git clone https://github.com/aws-solutions-library-samples/osml-data-intake.git
```
2. Run `tox` to create a virtual environment
```sh
cd osml-data-intake
tox
```
### Documentation
You can find documentation for this library in the `./doc` directory. Sphinx is used to construct a searchable HTML
version of the API documents.
```shell
tox -e docs
```
## Testing Your Setup
After setting up your environment, you can verify your setup by sending a test message to the SNS topic that will trigger your application workflow. This is useful for ensuring that your processing pipeline works correctly with a given image.
**Prerequisites:**
- Ensure that your AWS credentials are configured properly in the environment.
- Make sure that you have the AWS CLI installed and configured.
- Deploy the osml-data-intake infrastructure using the [guidance package](https://github.com/aws-solutions-library-samples/guidance-for-processing-overhead-imagery-on-aws)
**Run the Test Command:**
1. Replace the following with your specific details:
- **Topic ARN**: Update the `--topic-arn` argument with the ARN of the SNS topic that triggers your application.
- **S3 URL**: Replace the S3 URL in the `--message` argument with the URL of the image file you want to test.
2. Execute the following command, substituting your actual values:
```bash
python3 bin/local_cli.py --topic-arn <YOUR_TOPIC_ARN> --s3-uri <YOUR_S3_URI>
```
3. **Expected Output**:
- This will trigger the processing of the specified image file in your application.
- Verify that the auxiliary files are generated and uploaded to your configured S3 bucket, and ensure that the logs indicate a successful run.
## Support & Feedback
To post feedback, submit feature ideas, or report bugs, please use the [Issues](https://github.com/aws-solutions-library-samples/osml-data-intake/issues) section of this GitHub repo.
If you are interested in contributing to OversightML Data Intake, see the [CONTRIBUTING](https://github.com/aws-solutions-library-samples/osml-data-intake/CONTRIBUTING.md) guide.
## Security
See [CONTRIBUTING](https://github.com/aws-solutions-library-samples/osml-data-intake/CONTRIBUTING.md) for more information.
## License
MIT No Attribution Licensed. See [LICENSE](https://github.com/aws-solutions-library-samples/osml-data-intake/LICENSE).
Raw data
{
"_id": null,
"home_page": null,
"name": "osml-data-intake",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": "Amazon Web Services",
"author_email": "aws-osml-admin@amazon.com",
"download_url": "https://files.pythonhosted.org/packages/cc/ab/c55097a16dd362ef6ab2ad6deb7afa7b5b2b54d24e86f3331ea230ca5187/osml_data_intake-1.0.0.tar.gz",
"platform": null,
"description": "# OSML Data Intake\n\n## Overview\nThis application facilitates the processing, conversion, and management of satellite imagery metadata as part of the OversightML (OSML) framework and can be deployed as part of the OSML [guidance package](https://github.com/aws-solutions-library-samples/guidance-for-processing-overhead-imagery-on-aws). It leverages the GDAL library and integrates with Amazon S3 for seamless storage and sharing to provide imagery metadata to other service components. Below is an overview of the main features:\n\n### Image Metadata Processing\nExtracts and processes metadata from satellite imagery files, such as image dimensions and geographical coordinates. Generates auxiliary .aux and .ovr files for optimized image previews.\n\n### SpatioTemporal Asset Catalogs (STAC) Item Generation\nIncorporates converted coordinates into STAC-compliant data formats for indexing and sharing geospatial information. Automatically handles data formatting and polygon creation.\n\n### S3 Output\nUploads auxiliary files and metadata to Amazon S3 using the boto3 SDK. Logs success and error messages for uploads to aid in debugging and auditing.\n\n### Table of Contents\n* [Getting Started](#getting-started)\n * [Prerequisites](#prerequisites)\n * [Installation Guide](#installation-guide)\n * [Documentation](#documentation)\n* [Testing Your Setup](#testing-your-setup)\n* [Support & Feedback](#support--feedback)\n* [Security](#security)\n* [License](#license)\n\n\n## Getting Started\n### Prerequisites\n\nFirst, ensure you have installed the following tools locally\n\n1. [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)\n2. [docker](https://nodejs.org/en)\n3. [tox](https://tox.wiki/en/latest/installation.html)\n\n### Installation Guide\n\n1. Clone `osml-data-intake` package into your desktop\n\n```sh\ngit clone https://github.com/aws-solutions-library-samples/osml-data-intake.git\n```\n\n2. Run `tox` to create a virtual environment\n\n```sh\ncd osml-data-intake\ntox\n```\n\n### Documentation\n\nYou can find documentation for this library in the `./doc` directory. Sphinx is used to construct a searchable HTML\nversion of the API documents.\n\n```shell\ntox -e docs\n```\n\n## Testing Your Setup\n\nAfter setting up your environment, you can verify your setup by sending a test message to the SNS topic that will trigger your application workflow. This is useful for ensuring that your processing pipeline works correctly with a given image.\n\n**Prerequisites:**\n- Ensure that your AWS credentials are configured properly in the environment.\n- Make sure that you have the AWS CLI installed and configured.\n- Deploy the osml-data-intake infrastructure using the [guidance package](https://github.com/aws-solutions-library-samples/guidance-for-processing-overhead-imagery-on-aws)\n\n**Run the Test Command:**\n\n1. Replace the following with your specific details:\n - **Topic ARN**: Update the `--topic-arn` argument with the ARN of the SNS topic that triggers your application.\n - **S3 URL**: Replace the S3 URL in the `--message` argument with the URL of the image file you want to test.\n\n2. Execute the following command, substituting your actual values:\n\n ```bash\n python3 bin/local_cli.py --topic-arn <YOUR_TOPIC_ARN> --s3-uri <YOUR_S3_URI>\n ```\n\n3. **Expected Output**:\n - This will trigger the processing of the specified image file in your application.\n - Verify that the auxiliary files are generated and uploaded to your configured S3 bucket, and ensure that the logs indicate a successful run.\n\n\n## Support & Feedback\n\nTo post feedback, submit feature ideas, or report bugs, please use the [Issues](https://github.com/aws-solutions-library-samples/osml-data-intake/issues) section of this GitHub repo.\n\nIf you are interested in contributing to OversightML Data Intake, see the [CONTRIBUTING](https://github.com/aws-solutions-library-samples/osml-data-intake/CONTRIBUTING.md) guide.\n\n## Security\n\nSee [CONTRIBUTING](https://github.com/aws-solutions-library-samples/osml-data-intake/CONTRIBUTING.md) for more information.\n\n## License\n\nMIT No Attribution Licensed. See [LICENSE](https://github.com/aws-solutions-library-samples/osml-data-intake/LICENSE).\n",
"bugtrack_url": null,
"license": " MIT No Attribution Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
"summary": "Package to run the osml image intake component service.",
"version": "1.0.0",
"project_urls": {
"Source": "https://github.com/aws-solutions-library-samples/osml-data-intake",
"Tracker": "https://github.com/aws-solutions-library-samples/osml-data-intake/issues"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "20a771197130f9a77f4b1c168f1ca6004f3b910506165745fc4977071608c94a",
"md5": "cad03ac936fe1d27aeff9ba2ccc51243",
"sha256": "67c6e1da248af04a788f02898e1eaddc889547f1f3eeb48f30e56e1d45c3dffb"
},
"downloads": -1,
"filename": "osml_data_intake-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cad03ac936fe1d27aeff9ba2ccc51243",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 12253,
"upload_time": "2024-06-06T19:06:19",
"upload_time_iso_8601": "2024-06-06T19:06:19.079390Z",
"url": "https://files.pythonhosted.org/packages/20/a7/71197130f9a77f4b1c168f1ca6004f3b910506165745fc4977071608c94a/osml_data_intake-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ccabc55097a16dd362ef6ab2ad6deb7afa7b5b2b54d24e86f3331ea230ca5187",
"md5": "f3280e4d75cb048550c62d283b6ace32",
"sha256": "e2ced91e07157f0bc2f18208ec7156b000044593b6db9aa704e5b4b76a67bc41"
},
"downloads": -1,
"filename": "osml_data_intake-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "f3280e4d75cb048550c62d283b6ace32",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 12321,
"upload_time": "2024-06-06T19:06:20",
"upload_time_iso_8601": "2024-06-06T19:06:20.752346Z",
"url": "https://files.pythonhosted.org/packages/cc/ab/c55097a16dd362ef6ab2ad6deb7afa7b5b2b54d24e86f3331ea230ca5187/osml_data_intake-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-06 19:06:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "aws-solutions-library-samples",
"github_project": "osml-data-intake",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"tox": true,
"lcname": "osml-data-intake"
}
