b-aws-cdk-parallel


Nameb-aws-cdk-parallel JSON
Version 3.3.0 PyPI version JSON
download
home_pagehttps://github.com/biomapas/B.AwsCdkParallel.git
SummaryPackage that enables deployment of AWS CDK stacks in parallel.
upload_time2022-12-02 12:04:44
maintainer
docs_urlNone
authorLaimonas Sutkus
requires_python
licenseApache License 2.0
keywords aws iac cdk parallel
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # B.AwsCdkParallel

A python based package that enables AWS CDK parallel stack deployments.

### Description

One of the biggest downsides of AWS CDK IaC tool is the sequential deployments.
If you have many stacks within your project - it can take hours and hours till
everything gets deployed. Wouldn't it be cool to parallelize them? According to 
AWS CDK tool maintainers - they are not even thinking right now to include such 
functionality. Hence, this project was built. This project allows you to run 
traditional `cdk deploy *` and `cdk destroy * -f`. But the main trick is that it 
can do it in parallel - massively increasing the speed of your deployments.

### Remarks

[Biomapas](https://biomapas.com) aims to modernise life-science 
industry by sharing its IT knowledge with other companies and 
the community. This is an open source library intended to be used 
by anyone. Improvements and pull requests are welcome.

Some techniques and inspirations were taken from this blog post:<br>
https://taimos.de/blog/deploying-your-cdk-app-to-different-stages-and-environments

General issue is being discussed on github:<br>
https://github.com/aws/aws-cdk/issues/1973

### Related technology

- Python 3
- AWS CDK
- AWS CloudFormation

### Assumptions

The project assumes the following:

- You have basic-good knowledge in python programming.
- You have basic-good knowledge in AWS.
- You have very good knowledge in AWS CDK.

### Useful sources

- Read more AWS CDK:<br>
https://github.com/aws/aws-cdk

- Read more about parallel AWS CDK deployments:<br>
https://taimos.de/blog/deploying-your-cdk-app-to-different-stages-and-environments

### Install

The project is built and uploaded to PyPi. Install it by using pip.

```
pip install b_aws_cdk_parallel
```

Or directly install it through source.

```
pip install .
```

### Usage & Examples

#### Programmatic usage

The quickest and easiest example:

```python
from b_aws_cdk_parallel.deployment_executor import DeploymentExecutor
from b_aws_cdk_parallel.deployment_type import DeploymentType

executor = DeploymentExecutor(type=DeploymentType.DEPLOY)
executor.run()

executor = DeploymentExecutor(type=DeploymentType.DESTROY)
executor.run()
```

The more advanced example to deploy/destroy:

```python
from b_aws_cdk_parallel.deployment_executor import DeploymentExecutor
from b_aws_cdk_parallel.deployment_type import DeploymentType
from b_aws_cdk_parallel.cdk_arguments import CdkArguments

executor = DeploymentExecutor(
    type=DeploymentType.DEPLOY, # Or DESTROY
    # You can specify a full path to your CDK app.
    path='/optional/path/to/cdk/app',
    # You can specify OS-level global parameters.
    env={
        'optional': 'os-level environment variables'
    },
    # You can specify AWS-CDK-specific arguments.
    cdk_arguments=CdkArguments(
        aws_cdk_app_stacks_to_deploy=['MyCoolStack'],
        aws_cdk_app_parameters=['Test1=Test1'],
        aws_cdk_app_context=['Context1=Context1']
    )
)

executor.run()
```

The library generates beautiful stack dependency outputs for easier debugging:

```
----- Stack dependency graph: -----
» Stack1: []
× Stack2: [Stack1]
× Stack3: [Stack1]
× Stack4: [Stack1, Stack2, Stack3]
× Stack5: [Stack1, Stack4]
» Stack8: []
× Stack7: [Stack1, Stack2, Stack3, Stack4, Stack5]
× Stack6: [Stack1, Stack7]
× Stack10: [Stack1, Stack6, Stack7]
× Stack9: [Stack1, Stack8]
» B-Aws-Cdk-Parallel-MainStack-3: []


[Stack2]  Doing stuff...
[Stack2]  Doing stuff...
[Stack4]  Doing stuff...
[Stack3]  Doing stuff...
```

#### CLI usage

The library also exposes CLI access. 

To get usage help, simply run:

```
acdk -h
```

To deploy infrastructure, run:

```
acdk deploy --path /project/app/
```

To destroy infrastructure, run:

```
acdk destroy --path /project/app/
```

### Testing

This project has integration tests based on pytest. To run tests, simply run:

```
pytest b_aws_cdk_parallel_test/integration/tests
```

### Contribution

Found a bug? Want to add or suggest a new feature?<br>
Contributions of any kind are gladly welcome. You may contact us 
directly, create a pull-request or an issue in github platform.
Lets modernize the world together.


# Release history

### 3.3.0
* Add `--max-parallel-deployments` CLI flag to allow control of number the of stack deployments
  that run at the same time. For larger projects this can significantly reduce the amount of 
  memory used when deploying a highly parallel stack.

### 3.2.0
* Remove `aws-cdk.core` dependency as it is probably not needed (if the project is developed
  with AWS CDK anyways it will have `core` module installed). By removing this dependency both
  `V1` and `V2` versions of AWS CDK should work.

### 3.1.0
* Fix dependencies.
* Remove unnecessary dependencies.

### 3.0.0
* cdk v2 compatibility.
* Use the stack displayName when calling the cdk deploy command.

### 2.3.0
* Improve parallel stack deployments by introducing `--exclusively` flag.
* Make dependencies graph beautiful.
* Output beautiful stack names (not aws cdk generated names).
* Fix --path CLI bug.

### 2.2.0
* Support "--parameters".
* Support "--context".
* Support specifying stacks.
* Add a lot more tests.

### 2.1.0
* Expose CLI command `acdk` to enable deployment/destruction through CLI.

### 2.0.0
* Major bug fixes for resolving stack dependencies. The algorithm for parallel stack destruction has fundamentally changed.
* Added integration tests to test against an actual AWS environment.
* More improvements will come for 2.1.0.

### 1.3.0
* Do not rebuild assets on destroy.

### 1.2.0
* Do not rebuild assets on deployment.

### 1.1.0
* Add ability to control maximum parallel deployments.

### 1.0.0
* Complete rework of the project. Build a dependency tree to determine what to deploy.

### 0.4.1
* Raise exception in case of a failed deployment.

### 0.4.0
* Cdk list command should also receive path and environment.

### 0.3.0
* Add ability to specify CDK path and environment variables for processes.

### 0.2.0
* Add ability to retry main deployment too.

### 0.1.0
* Refactor project to make it more debug-friendly.

### 0.0.2
* Upgrade dependencies.

### 0.0.1
* Initial build.



            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/biomapas/B.AwsCdkParallel.git",
    "name": "b-aws-cdk-parallel",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "AWS IAC CDK Parallel",
    "author": "Laimonas Sutkus",
    "author_email": "laimonas.sutkus@biomapas.com",
    "download_url": "https://files.pythonhosted.org/packages/67/ca/99544c8928c82ce74f4e8cf5145594f90a7945f402eaac1760d767b3d086/b_aws_cdk_parallel-3.3.0.tar.gz",
    "platform": null,
    "description": "# B.AwsCdkParallel\n\nA python based package that enables AWS CDK parallel stack deployments.\n\n### Description\n\nOne of the biggest downsides of AWS CDK IaC tool is the sequential deployments.\nIf you have many stacks within your project - it can take hours and hours till\neverything gets deployed. Wouldn't it be cool to parallelize them? According to \nAWS CDK tool maintainers - they are not even thinking right now to include such \nfunctionality. Hence, this project was built. This project allows you to run \ntraditional `cdk deploy *` and `cdk destroy * -f`. But the main trick is that it \ncan do it in parallel - massively increasing the speed of your deployments.\n\n### Remarks\n\n[Biomapas](https://biomapas.com) aims to modernise life-science \nindustry by sharing its IT knowledge with other companies and \nthe community. This is an open source library intended to be used \nby anyone. Improvements and pull requests are welcome.\n\nSome techniques and inspirations were taken from this blog post:<br>\nhttps://taimos.de/blog/deploying-your-cdk-app-to-different-stages-and-environments\n\nGeneral issue is being discussed on github:<br>\nhttps://github.com/aws/aws-cdk/issues/1973\n\n### Related technology\n\n- Python 3\n- AWS CDK\n- AWS CloudFormation\n\n### Assumptions\n\nThe project assumes the following:\n\n- You have basic-good knowledge in python programming.\n- You have basic-good knowledge in AWS.\n- You have very good knowledge in AWS CDK.\n\n### Useful sources\n\n- Read more AWS CDK:<br>\nhttps://github.com/aws/aws-cdk\n\n- Read more about parallel AWS CDK deployments:<br>\nhttps://taimos.de/blog/deploying-your-cdk-app-to-different-stages-and-environments\n\n### Install\n\nThe project is built and uploaded to PyPi. Install it by using pip.\n\n```\npip install b_aws_cdk_parallel\n```\n\nOr directly install it through source.\n\n```\npip install .\n```\n\n### Usage & Examples\n\n#### Programmatic usage\n\nThe quickest and easiest example:\n\n```python\nfrom b_aws_cdk_parallel.deployment_executor import DeploymentExecutor\nfrom b_aws_cdk_parallel.deployment_type import DeploymentType\n\nexecutor = DeploymentExecutor(type=DeploymentType.DEPLOY)\nexecutor.run()\n\nexecutor = DeploymentExecutor(type=DeploymentType.DESTROY)\nexecutor.run()\n```\n\nThe more advanced example to deploy/destroy:\n\n```python\nfrom b_aws_cdk_parallel.deployment_executor import DeploymentExecutor\nfrom b_aws_cdk_parallel.deployment_type import DeploymentType\nfrom b_aws_cdk_parallel.cdk_arguments import CdkArguments\n\nexecutor = DeploymentExecutor(\n    type=DeploymentType.DEPLOY, # Or DESTROY\n    # You can specify a full path to your CDK app.\n    path='/optional/path/to/cdk/app',\n    # You can specify OS-level global parameters.\n    env={\n        'optional': 'os-level environment variables'\n    },\n    # You can specify AWS-CDK-specific arguments.\n    cdk_arguments=CdkArguments(\n        aws_cdk_app_stacks_to_deploy=['MyCoolStack'],\n        aws_cdk_app_parameters=['Test1=Test1'],\n        aws_cdk_app_context=['Context1=Context1']\n    )\n)\n\nexecutor.run()\n```\n\nThe library generates beautiful stack dependency outputs for easier debugging:\n\n```\n----- Stack dependency graph: -----\n\u00bb Stack1: []\n\u00d7 Stack2: [Stack1]\n\u00d7 Stack3: [Stack1]\n\u00d7 Stack4: [Stack1, Stack2, Stack3]\n\u00d7 Stack5: [Stack1, Stack4]\n\u00bb Stack8: []\n\u00d7 Stack7: [Stack1, Stack2, Stack3, Stack4, Stack5]\n\u00d7 Stack6: [Stack1, Stack7]\n\u00d7 Stack10: [Stack1, Stack6, Stack7]\n\u00d7 Stack9: [Stack1, Stack8]\n\u00bb B-Aws-Cdk-Parallel-MainStack-3: []\n\n\n[Stack2]  Doing stuff...\n[Stack2]  Doing stuff...\n[Stack4]  Doing stuff...\n[Stack3]  Doing stuff...\n```\n\n#### CLI usage\n\nThe library also exposes CLI access. \n\nTo get usage help, simply run:\n\n```\nacdk -h\n```\n\nTo deploy infrastructure, run:\n\n```\nacdk deploy --path /project/app/\n```\n\nTo destroy infrastructure, run:\n\n```\nacdk destroy --path /project/app/\n```\n\n### Testing\n\nThis project has integration tests based on pytest. To run tests, simply run:\n\n```\npytest b_aws_cdk_parallel_test/integration/tests\n```\n\n### Contribution\n\nFound a bug? Want to add or suggest a new feature?<br>\nContributions of any kind are gladly welcome. You may contact us \ndirectly, create a pull-request or an issue in github platform.\nLets modernize the world together.\n\n\n# Release history\n\n### 3.3.0\n* Add `--max-parallel-deployments` CLI flag to allow control of number the of stack deployments\n  that run at the same time. For larger projects this can significantly reduce the amount of \n  memory used when deploying a highly parallel stack.\n\n### 3.2.0\n* Remove `aws-cdk.core` dependency as it is probably not needed (if the project is developed\n  with AWS CDK anyways it will have `core` module installed). By removing this dependency both\n  `V1` and `V2` versions of AWS CDK should work.\n\n### 3.1.0\n* Fix dependencies.\n* Remove unnecessary dependencies.\n\n### 3.0.0\n* cdk v2 compatibility.\n* Use the stack displayName when calling the cdk deploy command.\n\n### 2.3.0\n* Improve parallel stack deployments by introducing `--exclusively` flag.\n* Make dependencies graph beautiful.\n* Output beautiful stack names (not aws cdk generated names).\n* Fix --path CLI bug.\n\n### 2.2.0\n* Support \"--parameters\".\n* Support \"--context\".\n* Support specifying stacks.\n* Add a lot more tests.\n\n### 2.1.0\n* Expose CLI command `acdk` to enable deployment/destruction through CLI.\n\n### 2.0.0\n* Major bug fixes for resolving stack dependencies. The algorithm for parallel stack destruction has fundamentally changed.\n* Added integration tests to test against an actual AWS environment.\n* More improvements will come for 2.1.0.\n\n### 1.3.0\n* Do not rebuild assets on destroy.\n\n### 1.2.0\n* Do not rebuild assets on deployment.\n\n### 1.1.0\n* Add ability to control maximum parallel deployments.\n\n### 1.0.0\n* Complete rework of the project. Build a dependency tree to determine what to deploy.\n\n### 0.4.1\n* Raise exception in case of a failed deployment.\n\n### 0.4.0\n* Cdk list command should also receive path and environment.\n\n### 0.3.0\n* Add ability to specify CDK path and environment variables for processes.\n\n### 0.2.0\n* Add ability to retry main deployment too.\n\n### 0.1.0\n* Refactor project to make it more debug-friendly.\n\n### 0.0.2\n* Upgrade dependencies.\n\n### 0.0.1\n* Initial build.\n\n\n",
    "bugtrack_url": null,
    "license": "Apache License 2.0",
    "summary": "Package that enables deployment of AWS CDK stacks in parallel.",
    "version": "3.3.0",
    "split_keywords": [
        "aws",
        "iac",
        "cdk",
        "parallel"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "f62709b10d665551231242a3b4557764",
                "sha256": "7a1797765286fa9ae2b22191b9a34f41df411c57092fcb99e16e7164572f7b56"
            },
            "downloads": -1,
            "filename": "b_aws_cdk_parallel-3.3.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "f62709b10d665551231242a3b4557764",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 31444,
            "upload_time": "2022-12-02T12:04:42",
            "upload_time_iso_8601": "2022-12-02T12:04:42.016878Z",
            "url": "https://files.pythonhosted.org/packages/ad/50/8a7cfaddff6bc77f977e8d94752c340c96f7ea58bda7d4f12d6029d418bb/b_aws_cdk_parallel-3.3.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "089cbfa092a96ebf87a818d74db64547",
                "sha256": "0169894c27dac1da1b02bf83a616a2965daa9cd1669291f8a66b2df6cff8eff5"
            },
            "downloads": -1,
            "filename": "b_aws_cdk_parallel-3.3.0.tar.gz",
            "has_sig": false,
            "md5_digest": "089cbfa092a96ebf87a818d74db64547",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 20952,
            "upload_time": "2022-12-02T12:04:44",
            "upload_time_iso_8601": "2022-12-02T12:04:44.064787Z",
            "url": "https://files.pythonhosted.org/packages/67/ca/99544c8928c82ce74f4e8cf5145594f90a7945f402eaac1760d767b3d086/b_aws_cdk_parallel-3.3.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2022-12-02 12:04:44",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "biomapas",
    "github_project": "B.AwsCdkParallel.git",
    "lcname": "b-aws-cdk-parallel"
}
        
Elapsed time: 0.02156s