clear-skies-aws


Nameclear-skies-aws JSON
Version 1.8.5 PyPI version JSON
download
home_pagehttps://github.com/cmancone/clearskies-aws
Summaryclearskies bindings for working in AWS
upload_time2024-03-20 10:13:59
maintainerNone
docs_urlNone
authorConor Mancone
requires_python<4.0,>=3.8
licenseMIT
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # clearskies-aws

clearskies bindings for working in AWS, which means additional:

 - backends (DynamoDB, SQS)
 - Secret/environment integrations (parameter store/secret manager)
 - DB connectivity via IAM auth
 - Contexts (ALB, HTTP API Gateway, Rest API Gateway, direct Lambda invocation, lambda+SQS)

# Installation, Documentation, and Usage

To install:

```
pip3 install clear-skies-aws
```

# Usage

Anytime you use a context from `clearskies-aws`, the default dependencies are adjust to:

 1. Provide `dynamo_db_backend` as an allowed backend
 2. Configure clearskies to use SSM Parameter Store as the default secrets "manager".

In both cases you must provide the AWS region for your resources, which you do by setting the `AWS_REGION` environment variable (either in an actual environment variable or in your `.env` file).

## Paramter Store

To use the SSM parameter store you just inject the `secrets` variable into your callables:

```
import clearskies_aws

def parameter_store_demo(secrets):
    return secrets.get('/path/to/parameter')

execute_demo_in_elb = clearskies_aws.contexts.lambda_elb(parameter_store_demo)

def lambda_handler(event, context):
    return execute_demo_in_elb(event, context)
```

Also, per default behavior, clearskies can fetch things from your secret manager if specified in your environment/.env file.  For instance, if your database password is stored in parameter store, then you can reference it from your `.env` file with a standard cursor backend:

```
db_host = "path-to-aws.rds"
db_username = "sql_username"
db_password = "secret://path/to/password/in/parameter/store"
db_database = "sql_database_name"
```

## Secret Manager

If desired, you can swap out the parameter store integration for secret manager.  Just remember that you can configure parameter store to fetch secrets from secret manager, so you might be best off doing that and sticking with the default parameter store integration.  Still, if you want to use secret manager, you just configure it in your application or context:

```
import clearskies_aws

def secret_manager_demo(secrets):
    return secrets.get('SecretARNFromSecretManager')

execute_demo_in_elb = clearskies_aws.contexts.lambda_elb(
    parameter_store_demo,
    bindings={'secrets': clearskies_aws.secrets.SecretManager},
)

def lambda_handler(event, context):
    return execute_demo_in_elb(event, context)
```

## Contexts

clearskies_aws adds the following contexts:

| name/import                                                 | Usage                             |
|-------------------------------------------------------------|-----------------------------------|
| `clearskies_aws.contexts.lambda_api_gateway`                | Lambdas behind a Rest API Gateway |
| `clearskies_aws.contexts.lambda_elb`                        | Lambdas behind a simple ELB/ALB   |
| `clearskies_aws.contexts.lambda_http_gateway`               | Lambdas behind an HTTP Gateway    |
| `clearskies_aws.contexts.lambda_inocation`                  | Lambdas invoked directly          |
| `clearskies_aws.contexts.lambda_sqs_standard_partial_batch` | Lambdas attached to an SQS queue  |

### Lambdas+SQS

Here's a simple example of using the Lambda+SQS context:

```
import clearskies_aws
import clearskies

def process_record(request_data):
    print(request_data)

process_sqs = clearskies_aws.contexts.lambda_sqs_standard_partial_batch(process_record)
def lambda_handler(event, context):
    process_sqs(event, context)
```

See [the AWS docs](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html).

Note that, unlike other contexts, the Lambda+SQS context really only works with a simple callable.  Routing and other handlers don't make much sense here.  Keep in mind that, before invoking the Lambda, AWS may batch up records together in arbitrary ways.  The context will take care of this and will invoke your callable once **for each record in the AWS event** - not once for the event.  `request_data` will be populated with the actual message for the event.  In addition, it assumes that a JSON message was sent to the queue, so `request_data` will be an object/list/etc, rather than a string.  This is intended to be used with [partial batching](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting).  Therefore, if your function raises an error, the context will catch it, return a failure response for the cooresponding message, and then continue processing any other messages in the batch.

## SQS Backend

To use the SQS backend just declare it for your model, set the table name to return the queue url, and execute a "create" operation to send data to the queue.  Note that the SQS backend is write-only: you can "create" records (resulting in a message being sent to the queue), but you can't read data back out.  The way the queue system in SQS works is just too different than a standard database for that to make sense in the context of a clearskies model.

```
#!/usr/bin/env python3
import clearskies_aws
import clearskies
from collections import OrderedDict

class Product(clearskies.Model):
    def __init__(self, sqs_backend, columns):
        super().__init__(sqs_backend, columns)

    @classmethod
    def table_name(cls):
        return 'https://sqs.us-east-1.amazonaws.com/123456789012/test-queue'

    def columns_configuration(self):
        return OrderedDict([
            clearskies.column_types.string('name'),
            clearskies.column_types.string('description'),
        ])

def send_to_queue(products):
    for i in range(10):
        products.create({
            'name': 'test',
            'description': str(i),
        })

cli = clearskies_aws.contexts.cli(
    send_to_queue,
    binding_classes=[Product],
)
if __name__ == '__main__':
    cli()
```

## IAM DB Auth

For non-serverless RDS databases, AWS supports login via IAM.  You have to provide a few additional details in your environment to make this work:

| name                     | value                                                                                                                 |
|--------------------------|-----------------------------------------------------------------------------------------------------------------------|
| `AWS_REGION`             | The region your database is in (e.g. `us-east-1`)                                                                     |
| `db_endpoint`            | The endpoint from your database (available in RDS)                                                                    |
| `db_username`            | The username to use to connect                                                                                        |
| `db_database`            | The name of the database                                                                                              |
| `ssl_ca_bundle_filename` | Path to the appropriate SSL bundle (see https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html) |

and then you have to enable it in your application/context configuration:

```
import clearskies_aws

def cursor_via_iamdb_auth(cursor):
    print('I connected successfully!')

execute_demo_in_elb = clearskies_aws.contexts.lambda_elb(
    cursor_via_iamdb_auth,
    additional_configs=[clearskies_aws.di.IAMDBAuth]
)

def lambda_handler(event, context):
    return execute_demo_in_elb(event, context)
```

Of course normally you wouldn't want to interact with it directly.  Adding `IAMDBAuth` to your `additional_configs` and setting up the necessary environemnt variables will be sufficient that any models that use the `cursor_backend` will connect via IAM DB Auth, rather than using hard-coded passwords.

## IAM DB Auth with SSM Bastion

Coming shortly

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/cmancone/clearskies-aws",
    "name": "clear-skies-aws",
    "maintainer": null,
    "docs_url": null,
    "requires_python": "<4.0,>=3.8",
    "maintainer_email": null,
    "keywords": null,
    "author": "Conor Mancone",
    "author_email": "cmancone@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/fd/72/100a5fb3c3f5b5e1c7613ab4c150ccae59508ed913747fa06ef3a0a37b30/clear_skies_aws-1.8.5.tar.gz",
    "platform": null,
    "description": "# clearskies-aws\n\nclearskies bindings for working in AWS, which means additional:\n\n - backends (DynamoDB, SQS)\n - Secret/environment integrations (parameter store/secret manager)\n - DB connectivity via IAM auth\n - Contexts (ALB, HTTP API Gateway, Rest API Gateway, direct Lambda invocation, lambda+SQS)\n\n# Installation, Documentation, and Usage\n\nTo install:\n\n```\npip3 install clear-skies-aws\n```\n\n# Usage\n\nAnytime you use a context from `clearskies-aws`, the default dependencies are adjust to:\n\n 1. Provide `dynamo_db_backend` as an allowed backend\n 2. Configure clearskies to use SSM Parameter Store as the default secrets \"manager\".\n\nIn both cases you must provide the AWS region for your resources, which you do by setting the `AWS_REGION` environment variable (either in an actual environment variable or in your `.env` file).\n\n## Paramter Store\n\nTo use the SSM parameter store you just inject the `secrets` variable into your callables:\n\n```\nimport clearskies_aws\n\ndef parameter_store_demo(secrets):\n    return secrets.get('/path/to/parameter')\n\nexecute_demo_in_elb = clearskies_aws.contexts.lambda_elb(parameter_store_demo)\n\ndef lambda_handler(event, context):\n    return execute_demo_in_elb(event, context)\n```\n\nAlso, per default behavior, clearskies can fetch things from your secret manager if specified in your environment/.env file.  For instance, if your database password is stored in parameter store, then you can reference it from your `.env` file with a standard cursor backend:\n\n```\ndb_host = \"path-to-aws.rds\"\ndb_username = \"sql_username\"\ndb_password = \"secret://path/to/password/in/parameter/store\"\ndb_database = \"sql_database_name\"\n```\n\n## Secret Manager\n\nIf desired, you can swap out the parameter store integration for secret manager.  Just remember that you can configure parameter store to fetch secrets from secret manager, so you might be best off doing that and sticking with the default parameter store integration.  Still, if you want to use secret manager, you just configure it in your application or context:\n\n```\nimport clearskies_aws\n\ndef secret_manager_demo(secrets):\n    return secrets.get('SecretARNFromSecretManager')\n\nexecute_demo_in_elb = clearskies_aws.contexts.lambda_elb(\n    parameter_store_demo,\n    bindings={'secrets': clearskies_aws.secrets.SecretManager},\n)\n\ndef lambda_handler(event, context):\n    return execute_demo_in_elb(event, context)\n```\n\n## Contexts\n\nclearskies_aws adds the following contexts:\n\n| name/import                                                 | Usage                             |\n|-------------------------------------------------------------|-----------------------------------|\n| `clearskies_aws.contexts.lambda_api_gateway`                | Lambdas behind a Rest API Gateway |\n| `clearskies_aws.contexts.lambda_elb`                        | Lambdas behind a simple ELB/ALB   |\n| `clearskies_aws.contexts.lambda_http_gateway`               | Lambdas behind an HTTP Gateway    |\n| `clearskies_aws.contexts.lambda_inocation`                  | Lambdas invoked directly          |\n| `clearskies_aws.contexts.lambda_sqs_standard_partial_batch` | Lambdas attached to an SQS queue  |\n\n### Lambdas+SQS\n\nHere's a simple example of using the Lambda+SQS context:\n\n```\nimport clearskies_aws\nimport clearskies\n\ndef process_record(request_data):\n    print(request_data)\n\nprocess_sqs = clearskies_aws.contexts.lambda_sqs_standard_partial_batch(process_record)\ndef lambda_handler(event, context):\n    process_sqs(event, context)\n```\n\nSee [the AWS docs](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html).\n\nNote that, unlike other contexts, the Lambda+SQS context really only works with a simple callable.  Routing and other handlers don't make much sense here.  Keep in mind that, before invoking the Lambda, AWS may batch up records together in arbitrary ways.  The context will take care of this and will invoke your callable once **for each record in the AWS event** - not once for the event.  `request_data` will be populated with the actual message for the event.  In addition, it assumes that a JSON message was sent to the queue, so `request_data` will be an object/list/etc, rather than a string.  This is intended to be used with [partial batching](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting).  Therefore, if your function raises an error, the context will catch it, return a failure response for the cooresponding message, and then continue processing any other messages in the batch.\n\n## SQS Backend\n\nTo use the SQS backend just declare it for your model, set the table name to return the queue url, and execute a \"create\" operation to send data to the queue.  Note that the SQS backend is write-only: you can \"create\" records (resulting in a message being sent to the queue), but you can't read data back out.  The way the queue system in SQS works is just too different than a standard database for that to make sense in the context of a clearskies model.\n\n```\n#!/usr/bin/env python3\nimport clearskies_aws\nimport clearskies\nfrom collections import OrderedDict\n\nclass Product(clearskies.Model):\n    def __init__(self, sqs_backend, columns):\n        super().__init__(sqs_backend, columns)\n\n    @classmethod\n    def table_name(cls):\n        return 'https://sqs.us-east-1.amazonaws.com/123456789012/test-queue'\n\n    def columns_configuration(self):\n        return OrderedDict([\n            clearskies.column_types.string('name'),\n            clearskies.column_types.string('description'),\n        ])\n\ndef send_to_queue(products):\n    for i in range(10):\n        products.create({\n            'name': 'test',\n            'description': str(i),\n        })\n\ncli = clearskies_aws.contexts.cli(\n    send_to_queue,\n    binding_classes=[Product],\n)\nif __name__ == '__main__':\n    cli()\n```\n\n## IAM DB Auth\n\nFor non-serverless RDS databases, AWS supports login via IAM.  You have to provide a few additional details in your environment to make this work:\n\n| name                     | value                                                                                                                 |\n|--------------------------|-----------------------------------------------------------------------------------------------------------------------|\n| `AWS_REGION`             | The region your database is in (e.g. `us-east-1`)                                                                     |\n| `db_endpoint`            | The endpoint from your database (available in RDS)                                                                    |\n| `db_username`            | The username to use to connect                                                                                        |\n| `db_database`            | The name of the database                                                                                              |\n| `ssl_ca_bundle_filename` | Path to the appropriate SSL bundle (see https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html) |\n\nand then you have to enable it in your application/context configuration:\n\n```\nimport clearskies_aws\n\ndef cursor_via_iamdb_auth(cursor):\n    print('I connected successfully!')\n\nexecute_demo_in_elb = clearskies_aws.contexts.lambda_elb(\n    cursor_via_iamdb_auth,\n    additional_configs=[clearskies_aws.di.IAMDBAuth]\n)\n\ndef lambda_handler(event, context):\n    return execute_demo_in_elb(event, context)\n```\n\nOf course normally you wouldn't want to interact with it directly.  Adding `IAMDBAuth` to your `additional_configs` and setting up the necessary environemnt variables will be sufficient that any models that use the `cursor_backend` will connect via IAM DB Auth, rather than using hard-coded passwords.\n\n## IAM DB Auth with SSM Bastion\n\nComing shortly\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "clearskies bindings for working in AWS",
    "version": "1.8.5",
    "project_urls": {
        "Homepage": "https://github.com/cmancone/clearskies-aws",
        "Repository": "https://github.com/cmancone/clearskies-aws"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "d1091268c96a729ccf2cf94686214876875d35d1621e789fc11c3d07b67832f5",
                "md5": "cf5ef92ddee522b97ddee8e7f85299d2",
                "sha256": "2342b22db4e31aa114719031f99594670133df0d2559ca7c3a39e83d890f5da1"
            },
            "downloads": -1,
            "filename": "clear_skies_aws-1.8.5-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "cf5ef92ddee522b97ddee8e7f85299d2",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": "<4.0,>=3.8",
            "size": 57718,
            "upload_time": "2024-03-20T10:13:57",
            "upload_time_iso_8601": "2024-03-20T10:13:57.741819Z",
            "url": "https://files.pythonhosted.org/packages/d1/09/1268c96a729ccf2cf94686214876875d35d1621e789fc11c3d07b67832f5/clear_skies_aws-1.8.5-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "fd72100a5fb3c3f5b5e1c7613ab4c150ccae59508ed913747fa06ef3a0a37b30",
                "md5": "5ffa277afc0154f3cb71eb5880fcfa42",
                "sha256": "85096e442fa3bf3fc587b7bd54d2c7cef0c291fd8eff10444e8d9b163fea178b"
            },
            "downloads": -1,
            "filename": "clear_skies_aws-1.8.5.tar.gz",
            "has_sig": false,
            "md5_digest": "5ffa277afc0154f3cb71eb5880fcfa42",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": "<4.0,>=3.8",
            "size": 40328,
            "upload_time": "2024-03-20T10:13:59",
            "upload_time_iso_8601": "2024-03-20T10:13:59.955143Z",
            "url": "https://files.pythonhosted.org/packages/fd/72/100a5fb3c3f5b5e1c7613ab4c150ccae59508ed913747fa06ef3a0a37b30/clear_skies_aws-1.8.5.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-03-20 10:13:59",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "cmancone",
    "github_project": "clearskies-aws",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "lcname": "clear-skies-aws"
}
        
Elapsed time: 0.23768s