# clearskies-aws
clearskies bindings for working in AWS, which means additional:
- backends (DynamoDB, SQS)
- Secret/environment integrations (parameter store/secret manager)
- DB connectivity via IAM auth
- Contexts (ALB, HTTP API Gateway, Rest API Gateway, direct Lambda invocation, lambda+SQS)
# Installation, Documentation, and Usage
To install:
```
pip3 install clear-skies-aws
```
# Usage
Anytime you use a context from `clearskies-aws`, the default dependencies are adjust to:
1. Provide `dynamo_db_backend` as an allowed backend
2. Configure clearskies to use SSM Parameter Store as the default secrets "manager".
In both cases you must provide the AWS region for your resources, which you do by setting the `AWS_REGION` environment variable (either in an actual environment variable or in your `.env` file).
## Paramter Store
To use the SSM parameter store you just inject the `secrets` variable into your callables:
```
import clearskies_aws
def parameter_store_demo(secrets):
return secrets.get('/path/to/parameter')
execute_demo_in_elb = clearskies_aws.contexts.lambda_elb(parameter_store_demo)
def lambda_handler(event, context):
return execute_demo_in_elb(event, context)
```
Also, per default behavior, clearskies can fetch things from your secret manager if specified in your environment/.env file. For instance, if your database password is stored in parameter store, then you can reference it from your `.env` file with a standard cursor backend:
```
db_host = "path-to-aws.rds"
db_username = "sql_username"
db_password = "secret://path/to/password/in/parameter/store"
db_database = "sql_database_name"
```
## Secret Manager
If desired, you can swap out the parameter store integration for secret manager. Just remember that you can configure parameter store to fetch secrets from secret manager, so you might be best off doing that and sticking with the default parameter store integration. Still, if you want to use secret manager, you just configure it in your application or context:
```
import clearskies_aws
def secret_manager_demo(secrets):
return secrets.get('SecretARNFromSecretManager')
execute_demo_in_elb = clearskies_aws.contexts.lambda_elb(
parameter_store_demo,
bindings={'secrets': clearskies_aws.secrets.SecretManager},
)
def lambda_handler(event, context):
return execute_demo_in_elb(event, context)
```
## Contexts
clearskies_aws adds the following contexts:
| name/import | Usage |
|-------------------------------------------------------------|-----------------------------------|
| `clearskies_aws.contexts.lambda_api_gateway` | Lambdas behind a Rest API Gateway |
| `clearskies_aws.contexts.lambda_elb` | Lambdas behind a simple ELB/ALB |
| `clearskies_aws.contexts.lambda_http_gateway` | Lambdas behind an HTTP Gateway |
| `clearskies_aws.contexts.lambda_inocation` | Lambdas invoked directly |
| `clearskies_aws.contexts.lambda_sqs_standard_partial_batch` | Lambdas attached to an SQS queue |
### Lambdas+SQS
Here's a simple example of using the Lambda+SQS context:
```
import clearskies_aws
import clearskies
def process_record(request_data):
print(request_data)
process_sqs = clearskies_aws.contexts.lambda_sqs_standard_partial_batch(process_record)
def lambda_handler(event, context):
process_sqs(event, context)
```
See [the AWS docs](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html).
Note that, unlike other contexts, the Lambda+SQS context really only works with a simple callable. Routing and other handlers don't make much sense here. Keep in mind that, before invoking the Lambda, AWS may batch up records together in arbitrary ways. The context will take care of this and will invoke your callable once **for each record in the AWS event** - not once for the event. `request_data` will be populated with the actual message for the event. In addition, it assumes that a JSON message was sent to the queue, so `request_data` will be an object/list/etc, rather than a string. This is intended to be used with [partial batching](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting). Therefore, if your function raises an error, the context will catch it, return a failure response for the cooresponding message, and then continue processing any other messages in the batch.
## SQS Backend
To use the SQS backend just declare it for your model, set the table name to return the queue url, and execute a "create" operation to send data to the queue. Note that the SQS backend is write-only: you can "create" records (resulting in a message being sent to the queue), but you can't read data back out. The way the queue system in SQS works is just too different than a standard database for that to make sense in the context of a clearskies model.
```
#!/usr/bin/env python3
import clearskies_aws
import clearskies
from collections import OrderedDict
class Product(clearskies.Model):
def __init__(self, sqs_backend, columns):
super().__init__(sqs_backend, columns)
@classmethod
def table_name(cls):
return 'https://sqs.us-east-1.amazonaws.com/123456789012/test-queue'
def columns_configuration(self):
return OrderedDict([
clearskies.column_types.string('name'),
clearskies.column_types.string('description'),
])
def send_to_queue(products):
for i in range(10):
products.create({
'name': 'test',
'description': str(i),
})
cli = clearskies_aws.contexts.cli(
send_to_queue,
binding_classes=[Product],
)
if __name__ == '__main__':
cli()
```
## IAM DB Auth
For non-serverless RDS databases, AWS supports login via IAM. You have to provide a few additional details in your environment to make this work:
| name | value |
|--------------------------|-----------------------------------------------------------------------------------------------------------------------|
| `AWS_REGION` | The region your database is in (e.g. `us-east-1`) |
| `db_endpoint` | The endpoint from your database (available in RDS) |
| `db_username` | The username to use to connect |
| `db_database` | The name of the database |
| `ssl_ca_bundle_filename` | Path to the appropriate SSL bundle (see https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html) |
and then you have to enable it in your application/context configuration:
```
import clearskies_aws
def cursor_via_iamdb_auth(cursor):
print('I connected successfully!')
execute_demo_in_elb = clearskies_aws.contexts.lambda_elb(
cursor_via_iamdb_auth,
additional_configs=[clearskies_aws.di.IAMDBAuth]
)
def lambda_handler(event, context):
return execute_demo_in_elb(event, context)
```
Of course normally you wouldn't want to interact with it directly. Adding `IAMDBAuth` to your `additional_configs` and setting up the necessary environemnt variables will be sufficient that any models that use the `cursor_backend` will connect via IAM DB Auth, rather than using hard-coded passwords.
## IAM DB Auth with SSM Bastion
Coming shortly
Raw data
{
"_id": null,
"home_page": "https://github.com/cmancone/clearskies-aws",
"name": "clear-skies-aws",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": null,
"author": "Conor Mancone",
"author_email": "cmancone@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/5a/93/3d78ba114649021869746bc5032dccd49892d337237e8678eff974036b3c/clear_skies_aws-1.9.11.tar.gz",
"platform": null,
"description": "# clearskies-aws\n\nclearskies bindings for working in AWS, which means additional:\n\n - backends (DynamoDB, SQS)\n - Secret/environment integrations (parameter store/secret manager)\n - DB connectivity via IAM auth\n - Contexts (ALB, HTTP API Gateway, Rest API Gateway, direct Lambda invocation, lambda+SQS)\n\n# Installation, Documentation, and Usage\n\nTo install:\n\n```\npip3 install clear-skies-aws\n```\n\n# Usage\n\nAnytime you use a context from `clearskies-aws`, the default dependencies are adjust to:\n\n 1. Provide `dynamo_db_backend` as an allowed backend\n 2. Configure clearskies to use SSM Parameter Store as the default secrets \"manager\".\n\nIn both cases you must provide the AWS region for your resources, which you do by setting the `AWS_REGION` environment variable (either in an actual environment variable or in your `.env` file).\n\n## Paramter Store\n\nTo use the SSM parameter store you just inject the `secrets` variable into your callables:\n\n```\nimport clearskies_aws\n\ndef parameter_store_demo(secrets):\n return secrets.get('/path/to/parameter')\n\nexecute_demo_in_elb = clearskies_aws.contexts.lambda_elb(parameter_store_demo)\n\ndef lambda_handler(event, context):\n return execute_demo_in_elb(event, context)\n```\n\nAlso, per default behavior, clearskies can fetch things from your secret manager if specified in your environment/.env file. For instance, if your database password is stored in parameter store, then you can reference it from your `.env` file with a standard cursor backend:\n\n```\ndb_host = \"path-to-aws.rds\"\ndb_username = \"sql_username\"\ndb_password = \"secret://path/to/password/in/parameter/store\"\ndb_database = \"sql_database_name\"\n```\n\n## Secret Manager\n\nIf desired, you can swap out the parameter store integration for secret manager. Just remember that you can configure parameter store to fetch secrets from secret manager, so you might be best off doing that and sticking with the default parameter store integration. Still, if you want to use secret manager, you just configure it in your application or context:\n\n```\nimport clearskies_aws\n\ndef secret_manager_demo(secrets):\n return secrets.get('SecretARNFromSecretManager')\n\nexecute_demo_in_elb = clearskies_aws.contexts.lambda_elb(\n parameter_store_demo,\n bindings={'secrets': clearskies_aws.secrets.SecretManager},\n)\n\ndef lambda_handler(event, context):\n return execute_demo_in_elb(event, context)\n```\n\n## Contexts\n\nclearskies_aws adds the following contexts:\n\n| name/import | Usage |\n|-------------------------------------------------------------|-----------------------------------|\n| `clearskies_aws.contexts.lambda_api_gateway` | Lambdas behind a Rest API Gateway |\n| `clearskies_aws.contexts.lambda_elb` | Lambdas behind a simple ELB/ALB |\n| `clearskies_aws.contexts.lambda_http_gateway` | Lambdas behind an HTTP Gateway |\n| `clearskies_aws.contexts.lambda_inocation` | Lambdas invoked directly |\n| `clearskies_aws.contexts.lambda_sqs_standard_partial_batch` | Lambdas attached to an SQS queue |\n\n### Lambdas+SQS\n\nHere's a simple example of using the Lambda+SQS context:\n\n```\nimport clearskies_aws\nimport clearskies\n\ndef process_record(request_data):\n print(request_data)\n\nprocess_sqs = clearskies_aws.contexts.lambda_sqs_standard_partial_batch(process_record)\ndef lambda_handler(event, context):\n process_sqs(event, context)\n```\n\nSee [the AWS docs](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html).\n\nNote that, unlike other contexts, the Lambda+SQS context really only works with a simple callable. Routing and other handlers don't make much sense here. Keep in mind that, before invoking the Lambda, AWS may batch up records together in arbitrary ways. The context will take care of this and will invoke your callable once **for each record in the AWS event** - not once for the event. `request_data` will be populated with the actual message for the event. In addition, it assumes that a JSON message was sent to the queue, so `request_data` will be an object/list/etc, rather than a string. This is intended to be used with [partial batching](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting). Therefore, if your function raises an error, the context will catch it, return a failure response for the cooresponding message, and then continue processing any other messages in the batch.\n\n## SQS Backend\n\nTo use the SQS backend just declare it for your model, set the table name to return the queue url, and execute a \"create\" operation to send data to the queue. Note that the SQS backend is write-only: you can \"create\" records (resulting in a message being sent to the queue), but you can't read data back out. The way the queue system in SQS works is just too different than a standard database for that to make sense in the context of a clearskies model.\n\n```\n#!/usr/bin/env python3\nimport clearskies_aws\nimport clearskies\nfrom collections import OrderedDict\n\nclass Product(clearskies.Model):\n def __init__(self, sqs_backend, columns):\n super().__init__(sqs_backend, columns)\n\n @classmethod\n def table_name(cls):\n return 'https://sqs.us-east-1.amazonaws.com/123456789012/test-queue'\n\n def columns_configuration(self):\n return OrderedDict([\n clearskies.column_types.string('name'),\n clearskies.column_types.string('description'),\n ])\n\ndef send_to_queue(products):\n for i in range(10):\n products.create({\n 'name': 'test',\n 'description': str(i),\n })\n\ncli = clearskies_aws.contexts.cli(\n send_to_queue,\n binding_classes=[Product],\n)\nif __name__ == '__main__':\n cli()\n```\n\n## IAM DB Auth\n\nFor non-serverless RDS databases, AWS supports login via IAM. You have to provide a few additional details in your environment to make this work:\n\n| name | value |\n|--------------------------|-----------------------------------------------------------------------------------------------------------------------|\n| `AWS_REGION` | The region your database is in (e.g. `us-east-1`) |\n| `db_endpoint` | The endpoint from your database (available in RDS) |\n| `db_username` | The username to use to connect |\n| `db_database` | The name of the database |\n| `ssl_ca_bundle_filename` | Path to the appropriate SSL bundle (see https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html) |\n\nand then you have to enable it in your application/context configuration:\n\n```\nimport clearskies_aws\n\ndef cursor_via_iamdb_auth(cursor):\n print('I connected successfully!')\n\nexecute_demo_in_elb = clearskies_aws.contexts.lambda_elb(\n cursor_via_iamdb_auth,\n additional_configs=[clearskies_aws.di.IAMDBAuth]\n)\n\ndef lambda_handler(event, context):\n return execute_demo_in_elb(event, context)\n```\n\nOf course normally you wouldn't want to interact with it directly. Adding `IAMDBAuth` to your `additional_configs` and setting up the necessary environemnt variables will be sufficient that any models that use the `cursor_backend` will connect via IAM DB Auth, rather than using hard-coded passwords.\n\n## IAM DB Auth with SSM Bastion\n\nComing shortly\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "clearskies bindings for working in AWS",
"version": "1.9.11",
"project_urls": {
"Homepage": "https://github.com/cmancone/clearskies-aws",
"Repository": "https://github.com/cmancone/clearskies-aws"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "93ab5377842d1c3b379a07cd0d2401af1690b8ec9fc4878d0b6ef440a1a6b9e1",
"md5": "031ffc4ac8565806d83cb11f42a64efa",
"sha256": "444250d7e450bafed286b32557f552d16be1a027fed9c73d5c7f00cf2398ae5e"
},
"downloads": -1,
"filename": "clear_skies_aws-1.9.11-py3-none-any.whl",
"has_sig": false,
"md5_digest": "031ffc4ac8565806d83cb11f42a64efa",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 62419,
"upload_time": "2024-08-10T10:58:58",
"upload_time_iso_8601": "2024-08-10T10:58:58.379272Z",
"url": "https://files.pythonhosted.org/packages/93/ab/5377842d1c3b379a07cd0d2401af1690b8ec9fc4878d0b6ef440a1a6b9e1/clear_skies_aws-1.9.11-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5a933d78ba114649021869746bc5032dccd49892d337237e8678eff974036b3c",
"md5": "76fd323c61e54947806c7c02aa118970",
"sha256": "9f764016562da7ebb0f877bbbe09c787f4f6b92d2c33046256938cd5bb8a57a4"
},
"downloads": -1,
"filename": "clear_skies_aws-1.9.11.tar.gz",
"has_sig": false,
"md5_digest": "76fd323c61e54947806c7c02aa118970",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 43578,
"upload_time": "2024-08-10T10:59:00",
"upload_time_iso_8601": "2024-08-10T10:59:00.409432Z",
"url": "https://files.pythonhosted.org/packages/5a/93/3d78ba114649021869746bc5032dccd49892d337237e8678eff974036b3c/clear_skies_aws-1.9.11.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-10 10:59:00",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "cmancone",
"github_project": "clearskies-aws",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "clear-skies-aws"
}
JSON
