| Name | otf-addons-aws JSON |
| Version |
24.16.0
JSON |
| download |
| home_page | None |
| Summary | Addons for opentaskpy, giving it the ability to push/pull via AWS S3, and pull variables from AWS SSM Parameter Store. |
| upload_time | 2024-04-15 13:16:38 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.11 |
| license | GPLv3 |
| keywords |
automation
task
framework
aws
s3
ssm
otf
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
[](https://pypi.org/project/otf-addons-aws/)
[](https://codecov.io/gh/adammcdonagh/otf-addons-aws)
[](https://github.com/adammcdonagh/otf-addons-aws/blob/master/LICENSE)
[](https://github.com/adammcdonagh/otf-addons-aws/issues)
[](https://github.com/adammcdonagh/otf-addons-aws/stargazers)
This repository contains addons to allow integration with AWS components via [Open Task Framework (OTF)](https://github.com/adammcdonagh/open-task-framework)
Open Task Framework (OTF) is a Python based framework to make it easy to run predefined file transfers and scripts/commands on remote machines.
These addons include several additional features:
- A new plugin for SSM Param Store to pull dynamic variables
- A new remotehandler to push/pull files via AWS S3
- A new remote handler to trigger AWS Lambda functions
# AWS Credentials
This package uses boto3 to communicate with AWS.
Credentials can be set via config using equivalently named variables alongside the protocol definition e.g;
```json
"protocol": {
"name": "opentaskpy.addons.aws.remotehandlers.s3.S3Transfer",
"access_key_id": "some_key",
"secret_access_key": "some_secret_key",
"assume_role_arn": "arn:aws:iam::000000000000:role/some_role",
"region_name": "eu-west-1"
}
```
If the standard AWS environment variables are set, then these will be used if not set elsewhere. Otherwise, if running from AWS, then the IAM role of the machine running OTF will be used.
# Transfers
Transfers are defined the same as a normal SSH based transfer.
As part of the upload, the `bucket-owner-full-control` ACL flag is applied to all files. This can be disabled by setting `disableBucketOwnerControlACL` to `true` in the `protocol` definition
### Supported features
- Plain file watch
- File watch/transfer with file size and age constraints
- `move` & `delete` post copy actions
- Touching empty files after transfer. e.g. `.fin` files used as completion flags
- Touching empty files as an execution
### Limitations
- No support for log watch
# Configuration
JSON configs for transfers can be defined as follows:
## Example File Watch Only
```json
"source": {
"bucket": "test-bucket",
"fileWatch": {
"timeout": 15,
"directory": "src",
"fileRegex": ".*\\.txt"
},
"protocol": {
"name": "opentaskpy.addons.aws.remotehandlers.s3.S3Transfer"
}
}
```
## Example S3 Download
```json
"source": {
"bucket": "some-bucket",
"directory": "src",
"fileRegex": ".*\\.txt",
"protocol": {
"name": "opentaskpy.addons.aws.remotehandlers.s3.S3Transfer"
}
}
```
## Example S3 Upload
```json
"destination": [
{
"bucket": "some-bucket",
"directory": "dest",
"protocol": {
"name": "opentaskpy.addons.aws.remotehandlers.s3.S3Transfer"
}
}
]
```
## Example S3 upload with flag files
```json
"destination": [
{
"bucket": "some-bucket",
"directory": "dest",
"flag": {
"fullPath": "dest/some_fin.flg"
}
"protocol": {
"name": "opentaskpy.addons.aws.remotehandlers.s3.S3Transfer"
}
}
]
```
# Executions
The Lambda remote handler allows AWS Lambda functions to be called. When provided with a `functionArn` the function will be called with no parameters. If there's a payload to pass in, use the `payload` attribute in the execution definition to specify a JSON object to pass into the function.
## Asynchronous vs Synchronous Execution
Lambda functions can be called with either an `invocationType` of `Event` (default if not specified) or `RequestResponse`.
`Event` is asynchronous, and tells the Lambda function to trigger, but does not check that it ran successfully. This means it's up to you to make sure that you have appropriate monitoring of your Lambda functions.
`RequestResponse` will block until the Lambda function either completes, or times out. Boto3 has a timeout of 60 seconds, so this cannot be used for long running functions (over 1 minute). This also causes issues when used in conjunction with batches and timeouts. Since the request blocks, the thread cannot be killed by the batch thread, meaning that it will block any further execution until 60 seconds after triggering the lambda function.
## Example S3 Execution touch flag file
```json
{
"type": "execution",
"bucket": "test-bucket",
"key": "test_key.flg",
"protocol": {
"name": "opentaskpy.addons.aws.remotehandlers.s3.S3Transfer"
}
}
```
## Example Lambda function call
```json
{
"type": "execution",
"functionArn": "arn:aws:lambda:eu-west-1:000000000000:function:my-function",
"invocationType": "Event",
"payload": {
"file-name": "some_file.txt"
},
"protocol": {
"name": "opentaskpy.addons.aws.remotehandlers.lambda.LambdaExecution"
}
}
```
Raw data
{
"_id": null,
"home_page": null,
"name": "otf-addons-aws",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "automation, task, framework, aws, s3, ssm, otf",
"author": null,
"author_email": "Adam McDonagh <adam@elitemonkey.net>",
"download_url": "https://files.pythonhosted.org/packages/af/64/7a300c32139ddb42e3a8b1c4c2ac06a722f4d14083a48506dec0976da010/otf_addons_aws-24.16.0.tar.gz",
"platform": null,
"description": "[](https://pypi.org/project/otf-addons-aws/)\n\n[](https://codecov.io/gh/adammcdonagh/otf-addons-aws)\n[](https://github.com/adammcdonagh/otf-addons-aws/blob/master/LICENSE)\n[](https://github.com/adammcdonagh/otf-addons-aws/issues)\n[](https://github.com/adammcdonagh/otf-addons-aws/stargazers)\n\nThis repository contains addons to allow integration with AWS components via [Open Task Framework (OTF)](https://github.com/adammcdonagh/open-task-framework)\n\nOpen Task Framework (OTF) is a Python based framework to make it easy to run predefined file transfers and scripts/commands on remote machines.\n\nThese addons include several additional features:\n\n- A new plugin for SSM Param Store to pull dynamic variables\n- A new remotehandler to push/pull files via AWS S3\n- A new remote handler to trigger AWS Lambda functions\n\n# AWS Credentials\n\nThis package uses boto3 to communicate with AWS.\n\nCredentials can be set via config using equivalently named variables alongside the protocol definition e.g;\n\n```json\n\"protocol\": {\n \"name\": \"opentaskpy.addons.aws.remotehandlers.s3.S3Transfer\",\n \"access_key_id\": \"some_key\",\n \"secret_access_key\": \"some_secret_key\",\n \"assume_role_arn\": \"arn:aws:iam::000000000000:role/some_role\",\n \"region_name\": \"eu-west-1\"\n}\n```\n\nIf the standard AWS environment variables are set, then these will be used if not set elsewhere. Otherwise, if running from AWS, then the IAM role of the machine running OTF will be used.\n\n# Transfers\n\nTransfers are defined the same as a normal SSH based transfer.\n\nAs part of the upload, the `bucket-owner-full-control` ACL flag is applied to all files. This can be disabled by setting `disableBucketOwnerControlACL` to `true` in the `protocol` definition\n\n### Supported features\n\n- Plain file watch\n- File watch/transfer with file size and age constraints\n- `move` & `delete` post copy actions\n- Touching empty files after transfer. e.g. `.fin` files used as completion flags\n- Touching empty files as an execution\n\n### Limitations\n\n- No support for log watch\n\n# Configuration\n\nJSON configs for transfers can be defined as follows:\n\n## Example File Watch Only\n\n```json\n\"source\": {\n \"bucket\": \"test-bucket\",\n \"fileWatch\": {\n \"timeout\": 15,\n \"directory\": \"src\",\n \"fileRegex\": \".*\\\\.txt\"\n },\n \"protocol\": {\n \"name\": \"opentaskpy.addons.aws.remotehandlers.s3.S3Transfer\"\n }\n}\n```\n\n## Example S3 Download\n\n```json\n\"source\": {\n \"bucket\": \"some-bucket\",\n \"directory\": \"src\",\n \"fileRegex\": \".*\\\\.txt\",\n \"protocol\": {\n \"name\": \"opentaskpy.addons.aws.remotehandlers.s3.S3Transfer\"\n }\n}\n```\n\n## Example S3 Upload\n\n```json\n\"destination\": [\n {\n \"bucket\": \"some-bucket\",\n \"directory\": \"dest\",\n \"protocol\": {\n \"name\": \"opentaskpy.addons.aws.remotehandlers.s3.S3Transfer\"\n }\n }\n]\n```\n\n## Example S3 upload with flag files\n\n```json\n\"destination\": [\n {\n \"bucket\": \"some-bucket\",\n \"directory\": \"dest\",\n \"flag\": {\n \"fullPath\": \"dest/some_fin.flg\"\n }\n \"protocol\": {\n \"name\": \"opentaskpy.addons.aws.remotehandlers.s3.S3Transfer\"\n }\n }\n]\n```\n\n# Executions\n\nThe Lambda remote handler allows AWS Lambda functions to be called. When provided with a `functionArn` the function will be called with no parameters. If there's a payload to pass in, use the `payload` attribute in the execution definition to specify a JSON object to pass into the function.\n\n## Asynchronous vs Synchronous Execution\n\nLambda functions can be called with either an `invocationType` of `Event` (default if not specified) or `RequestResponse`.\n\n`Event` is asynchronous, and tells the Lambda function to trigger, but does not check that it ran successfully. This means it's up to you to make sure that you have appropriate monitoring of your Lambda functions.\n\n`RequestResponse` will block until the Lambda function either completes, or times out. Boto3 has a timeout of 60 seconds, so this cannot be used for long running functions (over 1 minute). This also causes issues when used in conjunction with batches and timeouts. Since the request blocks, the thread cannot be killed by the batch thread, meaning that it will block any further execution until 60 seconds after triggering the lambda function.\n\n## Example S3 Execution touch flag file\n\n```json\n{\n \"type\": \"execution\",\n \"bucket\": \"test-bucket\",\n \"key\": \"test_key.flg\",\n \"protocol\": {\n \"name\": \"opentaskpy.addons.aws.remotehandlers.s3.S3Transfer\"\n }\n}\n```\n\n## Example Lambda function call\n\n```json\n{\n \"type\": \"execution\",\n \"functionArn\": \"arn:aws:lambda:eu-west-1:000000000000:function:my-function\",\n \"invocationType\": \"Event\",\n \"payload\": {\n \"file-name\": \"some_file.txt\"\n },\n \"protocol\": {\n \"name\": \"opentaskpy.addons.aws.remotehandlers.lambda.LambdaExecution\"\n }\n}\n```\n",
"bugtrack_url": null,
"license": "GPLv3",
"summary": "Addons for opentaskpy, giving it the ability to push/pull via AWS S3, and pull variables from AWS SSM Parameter Store.",
"version": "24.16.0",
"project_urls": {
"Bug Tracker": "https://github.com/adammcdonagh/otf-addons-aws/issues",
"Changelog": "https://github.com/adammcdonagh/otf-addons-aws/blob/main/CHANGELOG.md",
"Homepage": "https://github.com/adammcdonagh/otf-addons-aws"
},
"split_keywords": [
"automation",
" task",
" framework",
" aws",
" s3",
" ssm",
" otf"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "af647a300c32139ddb42e3a8b1c4c2ac06a722f4d14083a48506dec0976da010",
"md5": "42e6c587d7c894acbb417cedc6099e27",
"sha256": "781db80aa79dd847502ab55af60ece6da28975a635442a9f0711350dea55a349"
},
"downloads": -1,
"filename": "otf_addons_aws-24.16.0.tar.gz",
"has_sig": false,
"md5_digest": "42e6c587d7c894acbb417cedc6099e27",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 43994,
"upload_time": "2024-04-15T13:16:38",
"upload_time_iso_8601": "2024-04-15T13:16:38.545288Z",
"url": "https://files.pythonhosted.org/packages/af/64/7a300c32139ddb42e3a8b1c4c2ac06a722f4d14083a48506dec0976da010/otf_addons_aws-24.16.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-15 13:16:38",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "adammcdonagh",
"github_project": "otf-addons-aws",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "otf-addons-aws"
}
