<img src="https://github.com/dbsystel/cdk-sops-secrets/blob/main/img/banner-dl-small.png?raw=true">
<p/>
[](https://github.com/dbsystel/cdk-sops-secrets/actions/workflows/release.yml)<br>
[](https://constructs.dev/packages/cdk-sops-secrets)<br>
[](https://www.npmjs.com/package/cdk-sops-secrets)
[](https://www.npmjs.com/package/cdk-sops-secrets)<br>
[](https://pypi.org/project/cdk-sops-secrets)
[](https://pypi.org/project/cdk-sops-secrets)<br>
[](https://codecov.io/gh/dbsystel/cdk-sops-secrets)
[](https://github.com/dbsystel/cdk-sops-secrets/issues?q=is%3Aissue+is%3Aopen+label%3A%22security+vulnerability%22)
## Introduction
This construct library provides a replacement for CDK SecretsManager secrets, with extended functionality for Mozilla/sops.
<p/><center><img src="img/flow.drawio.svg"></center><p/>
Using this library it is possible to populate Secrets with values from a Mozilla/sops file without additional scripts and steps in the CI stage. Thereby transformations like JSON conversion of YAML files and transformation into a flat, JSONPath like structure will be performed, but can be disabled.
Secrets filled in this way can be used immediately within the CloudFormation stack and dynamic references. This construct should handle all dependencies, if you use the `secretValueFromJson()` or `secretValue()` call to access secret values.
This way, secrets can be securely stored in git repositories and easily synchronized into AWS SecretsManager secrets.
## Stability
You can consider this package as stable. Updates will follow [Semantic Versioning](https://semver.org/).<br>
Nevertheless, I would recommend pinning the exact version of this library in your `package.json`.
## Prerequisites
* [AWS](https://aws.amazon.com/): I think you already knew it, but this construct will only work with an AWS account.
* [KMS Key](https://aws.amazon.com/kms/?nc1=h_ls): It makes most sense to encrypt your secrets with AWS KMS if you want to sync and use the secret content afterwards in your AWS account.
* [mozilla/sops](https://github.com/mozilla/sops): This construct assumes that you store your secrets encrypted via sops in your git repository.
* [CDK](https://aws.amazon.com/cdk/?nc1=h_ls): As this is a CDK construct, it's only useful if you use the CloudDevelopmentToolkit.
## Getting started
1. Create a Mozilla/sops secrets file (encrypted with an already existing KMS key) and place it somewhere in your git repository
2. Create a secret with the SopsSecret construct inside your app
```python
const secret = new SopsSecret(stack, 'SopsComplexSecretJSON', {
sopsFilePath: 'secets/sopsfile-encrypted.json',
});
```
3. Optional: Access the secret via dynamic references
```python
secret.secretValueFromJson('json.path.dotted.notation.accessor[0]').toString(),
```
## Advanced configuration examples
Even if using the main functionality should be done in 3 lines of code, there are more options to configure the constructs of this library. If you want to get an Overview of all available configuration options take a look at the [documentation at the CDK ConstructHub](https://constructs.dev/packages/cdk-sops-secrets).
The most useful settings will be explained in the further chapters:
### Getting a specific (older version)
While creating the secret or updating the entries of a secret, the native CDK function `cdk.FileSystem.fingerprint(...)` is used to generate the version information of the AWS SecretsManager secret.
Therefore, it is possible to reference the entries from a specific AWS SecretsManager version.
```python
const versionId = cdk.FileSystem.fingerprint(`./sops/SomeSecrets.json`)
const passphrase = ecs.Secret.fromSecretsManagerVersion(secretMgmt, { versionId: versionId }, 'MY_PRIVATE_PASSPHRASE')
const container = TaskDef.addContainer('Container', {
secrets: {
MY_PRIVATE_PASSPHRASE: passphrase,
},
});
```
### Default conversions and how to disable them?
As default behavior, the SopsSecret (via the SopsSync) will convert all content to JSON and flatten its structure. This is useful, because the AWS SecretsManager has some limitations if it comes to YAML and/or complex objects and decimal values. Even if you can store YAML, complex objects and even binaries in AWS SecretsManager secrets, you can't access their values via the SecretsManager API — you can only return them as is. So accessing (nested) values or values from YAML files won't be possible via [dynamic references](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html) in CloudFormation (and CDK). That's why I decided that conversion to JSON, flatten the structure and stringify all values should be the default behavior. But you can turn off all of these conversion steps:
```python
const secret = new SopsSecret(this, 'SopsComplexSecretJSON', {
convertToJSON: false, // disable converting the encrypted content to JSON
stringify: false, // disable stringifying all values
flatten: false, // disable flattening of the object structure
sopsFilePath: 'secrets/sopsfile-encrypted.json',
});
```
### Resource provider is missing permissions
Sometimes it can be necessary to access the IAM role of the SopsSync provider. If this is the case, you should create the provider before creating the SopsSecret, and pass the provider to it like this:
```python
// Create the provider
const provider = new SopsSyncProvider(this, 'CustomSopsSyncProvider');
// Grant whatever you need to the provider
const myExtraKmsKey = Key.fromKeyArn(this, 'MyExtraKmsKey', 'YourKeyArn');
myExtraKmsKey.grantDecrypt(provider);
// create the secret and pass the the provider to it
const secret = new SopsSecret(this, 'SopsComplexSecretJSON', {
sopsProvider: provider,
secretName: 'myCoolSecret',
sopsFilePath: 'secrets/sopsfile-encrypted.json',
});
```
### UploadType: INLINE / ASSET
I decided, that the default behavior should be "INLINE" because of the following consideration:
* Fewer permissions: If we use inline content instead of a S3 asset, the SopsSyncProvider does not need permissions to access the asset bucket and its KMS key.
* Faster: If we don't have to upload and download things from and to S3, it should be a little faster.
* Interchangeable: As we use the same information to generate the version of the secret, no new version of the secret should be created, if you change from INLINE to ASSET or vice versa, even if the CloudFormation resource updates.
* I personally think sops files are not that big, that we should run into limits, but if so — we can change to asset `uploadType`.
You can change the uplaodType via the properties:
```python
const secret = new SopsSecret(this, 'SopsWithAssetUpload', {
sopsFilePath: 'secrets/sopsfile-encrypted.json',
uploadType: UploadType.ASSET, // instead of the default UploadType.INLINE
});
```
## FAQ
### It does not work, what can I do?
Even if this construct has some unit and integration tests performed, there can be bugs and issues. As everything is performed by a cloudformation custom resource provider, a good starting point is the log of the corresponding lambda function. It should be located in your AWS Account under Cloudwatch -> Log groups:
`/aws/lambda/<YOUR-STACK-NAME>-SingletonLambdaSopsSyncProvider<SOMETHINGsomething1234>`
### Error getting data key: 0 successful groups required, got 0
This error message (and failed sync) is related to the mozilla/sops issues [#948](https://github.com/mozilla/sops/issues/948) and [#634](https://github.com/mozilla/sops/issues/634). You must not create your secret with the `--aws-profile` flag. This profile will be written to your sops filed and is required in every runtime environment. You have to define the profile to use via the environment variable `AWS_PROFILE` instead, to avoid this.
### Asset of sync lambda not found
The lambda asset code is generated relative to the path of the index.ts in this package. With tools like nx this can lead to wrong results, so that the asset could not be found.
You can override the asset path via the [cdk.json](https://docs.aws.amazon.com/cdk/v2/guide/get_context_var.html) or via the flag `-c`of the cdk cli.
The context used for this override is `sops_sync_provider_asset_path`.
So for example you can use
```bash
cdk deploy -c "sops_sync_provider_asset_path=some/path/asset.zip"
```
or in your cdk.json
```json
{
"context": {
"sops_sync_provider_asset_path": "some/path/asset.zip"
}
}
```
## Motivation
I have created this project to solve a recurring problem of syncing Mozilla/sops secrets into AWS SecretsManager in a convenient, secure way.
Other than that, or perhaps more importantly, my goal was to learn new things:
* Write a Golang lambda
* Writing unit tests incl. mocks in Golang
* Reproducible builds of Golang binaries (byte-by-byte identical)
* Build reproducible zips (byte-by-byte identical)
* Release a NPM package
* Setting up projects with projen
* CI/CD with GitHub actions
* CDK unit and integration tests
## Other Tools like this
The problem this Construct addresses is so good, already two other implementations exist:
* [isotoma/sops-secretsmanager-cdk](https://github.com/isotoma/sops-secretsmanager-cdk): Does nearly the same. Uses CustomResource, wraps the sops CLI, does not support flatten. Found it after I published my solution to NPM :-/
* [taimos/secretsmanager-versioning](https://github.com/taimos/secretsmanager-versioning): Different approach on the same problem. This is a CLI tool with very nice integration into CDK and also handles git versioning information.
## License
The Apache-2.0 license. Please have a look at the [LICENSE](LICENSE) and [LICENSE-3RD-PARTY](LICENSE-3RD-PARTY).
Raw data
{
"_id": null,
"home_page": "https://constructs.dev/packages/cdk-sops-secrets",
"name": "cdk-sops-secrets",
"maintainer": null,
"docs_url": null,
"requires_python": "~=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Markus Siebert<markus.siebert@deutschebahn.com>",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/48/ed/37f2d2b8da047edc63ffdf4c0da195aaf8cee32448925497cdd394293054/cdk-sops-secrets-1.6.16.tar.gz",
"platform": null,
"description": "<img src=\"https://github.com/dbsystel/cdk-sops-secrets/blob/main/img/banner-dl-small.png?raw=true\">\n<p/>\n\n\u00a0\n[](https://github.com/dbsystel/cdk-sops-secrets/actions/workflows/release.yml)<br>\n\n[](https://constructs.dev/packages/cdk-sops-secrets)<br>\n[](https://www.npmjs.com/package/cdk-sops-secrets)\u00a0\n[](https://www.npmjs.com/package/cdk-sops-secrets)<br>\n[](https://pypi.org/project/cdk-sops-secrets)\u00a0\n[](https://pypi.org/project/cdk-sops-secrets)<br>\n\n[](https://codecov.io/gh/dbsystel/cdk-sops-secrets)\u00a0\u00a0\n[](https://github.com/dbsystel/cdk-sops-secrets/issues?q=is%3Aissue+is%3Aopen+label%3A%22security+vulnerability%22)\u00a0\n\n## Introduction\n\nThis construct library provides a replacement for CDK SecretsManager secrets, with extended functionality for Mozilla/sops.\n\n<p/><center><img src=\"img/flow.drawio.svg\"></center><p/>\nUsing this library it is possible to populate Secrets with values from a Mozilla/sops file without additional scripts and steps in the CI stage. Thereby transformations like JSON conversion of YAML files and transformation into a flat, JSONPath like structure will be performed, but can be disabled.\n\nSecrets filled in this way can be used immediately within the CloudFormation stack and dynamic references. This construct should handle all dependencies, if you use the `secretValueFromJson()` or `secretValue()` call to access secret values.\n\nThis way, secrets can be securely stored in git repositories and easily synchronized into AWS SecretsManager secrets.\n\n## Stability\n\nYou can consider this package as stable. Updates will follow [Semantic Versioning](https://semver.org/).<br>\nNevertheless, I would recommend pinning the exact version of this library in your `package.json`.\n\n## Prerequisites\n\n* [AWS](https://aws.amazon.com/): I think you already knew it, but this construct will only work with an AWS account.\n\n* [KMS Key](https://aws.amazon.com/kms/?nc1=h_ls): It makes most sense to encrypt your secrets with AWS KMS if you want to sync and use the secret content afterwards in your AWS account.\n* [mozilla/sops](https://github.com/mozilla/sops): This construct assumes that you store your secrets encrypted via sops in your git repository.\n* [CDK](https://aws.amazon.com/cdk/?nc1=h_ls): As this is a CDK construct, it's only useful if you use the CloudDevelopmentToolkit.\n\n## Getting started\n\n1. Create a Mozilla/sops secrets file (encrypted with an already existing KMS key) and place it somewhere in your git repository\n2. Create a secret with the SopsSecret construct inside your app\n\n ```python\n const secret = new SopsSecret(stack, 'SopsComplexSecretJSON', {\n sopsFilePath: 'secets/sopsfile-encrypted.json',\n });\n ```\n3. Optional: Access the secret via dynamic references\n\n ```python\n secret.secretValueFromJson('json.path.dotted.notation.accessor[0]').toString(),\n ```\n\n## Advanced configuration examples\n\nEven if using the main functionality should be done in 3 lines of code, there are more options to configure the constructs of this library. If you want to get an Overview of all available configuration options take a look at the [documentation at the CDK ConstructHub](https://constructs.dev/packages/cdk-sops-secrets).\n\nThe most useful settings will be explained in the further chapters:\n\n### Getting a specific (older version)\n\nWhile creating the secret or updating the entries of a secret, the native CDK function `cdk.FileSystem.fingerprint(...)` is used to generate the version information of the AWS SecretsManager secret.\nTherefore, it is possible to reference the entries from a specific AWS SecretsManager version.\n\n```python\nconst versionId = cdk.FileSystem.fingerprint(`./sops/SomeSecrets.json`)\nconst passphrase = ecs.Secret.fromSecretsManagerVersion(secretMgmt, { versionId: versionId }, 'MY_PRIVATE_PASSPHRASE')\n\nconst container = TaskDef.addContainer('Container', {\n secrets: {\n MY_PRIVATE_PASSPHRASE: passphrase,\n },\n});\n```\n\n### Default conversions and how to disable them?\n\nAs default behavior, the SopsSecret (via the SopsSync) will convert all content to JSON and flatten its structure. This is useful, because the AWS SecretsManager has some limitations if it comes to YAML and/or complex objects and decimal values. Even if you can store YAML, complex objects and even binaries in AWS SecretsManager secrets, you can't access their values via the SecretsManager API \u2014 you can only return them as is. So accessing (nested) values or values from YAML files won't be possible via [dynamic references](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html) in CloudFormation (and CDK). That's why I decided that conversion to JSON, flatten the structure and stringify all values should be the default behavior. But you can turn off all of these conversion steps:\n\n```python\nconst secret = new SopsSecret(this, 'SopsComplexSecretJSON', {\n convertToJSON: false, // disable converting the encrypted content to JSON\n stringify: false, // disable stringifying all values\n flatten: false, // disable flattening of the object structure\n sopsFilePath: 'secrets/sopsfile-encrypted.json',\n});\n```\n\n### Resource provider is missing permissions\n\nSometimes it can be necessary to access the IAM role of the SopsSync provider. If this is the case, you should create the provider before creating the SopsSecret, and pass the provider to it like this:\n\n```python\n// Create the provider\nconst provider = new SopsSyncProvider(this, 'CustomSopsSyncProvider');\n// Grant whatever you need to the provider\nconst myExtraKmsKey = Key.fromKeyArn(this, 'MyExtraKmsKey', 'YourKeyArn');\nmyExtraKmsKey.grantDecrypt(provider);\n// create the secret and pass the the provider to it\nconst secret = new SopsSecret(this, 'SopsComplexSecretJSON', {\n sopsProvider: provider,\n secretName: 'myCoolSecret',\n sopsFilePath: 'secrets/sopsfile-encrypted.json',\n});\n```\n\n### UploadType: INLINE / ASSET\n\nI decided, that the default behavior should be \"INLINE\" because of the following consideration:\n\n* Fewer permissions: If we use inline content instead of a S3 asset, the SopsSyncProvider does not need permissions to access the asset bucket and its KMS key.\n* Faster: If we don't have to upload and download things from and to S3, it should be a little faster.\n* Interchangeable: As we use the same information to generate the version of the secret, no new version of the secret should be created, if you change from INLINE to ASSET or vice versa, even if the CloudFormation resource updates.\n* I personally think sops files are not that big, that we should run into limits, but if so \u2014 we can change to asset `uploadType`.\n\nYou can change the uplaodType via the properties:\n\n```python\nconst secret = new SopsSecret(this, 'SopsWithAssetUpload', {\n sopsFilePath: 'secrets/sopsfile-encrypted.json',\n uploadType: UploadType.ASSET, // instead of the default UploadType.INLINE\n});\n```\n\n## FAQ\n\n### It does not work, what can I do?\n\nEven if this construct has some unit and integration tests performed, there can be bugs and issues. As everything is performed by a cloudformation custom resource provider, a good starting point is the log of the corresponding lambda function. It should be located in your AWS Account under Cloudwatch -> Log groups:\n\n`/aws/lambda/<YOUR-STACK-NAME>-SingletonLambdaSopsSyncProvider<SOMETHINGsomething1234>`\n\n### Error getting data key: 0 successful groups required, got 0\n\nThis error message (and failed sync) is related to the mozilla/sops issues [#948](https://github.com/mozilla/sops/issues/948) and [#634](https://github.com/mozilla/sops/issues/634). You must not create your secret with the `--aws-profile` flag. This profile will be written to your sops filed and is required in every runtime environment. You have to define the profile to use via the environment variable `AWS_PROFILE` instead, to avoid this.\n\n### Asset of sync lambda not found\n\nThe lambda asset code is generated relative to the path of the index.ts in this package. With tools like nx this can lead to wrong results, so that the asset could not be found.\n\nYou can override the asset path via the [cdk.json](https://docs.aws.amazon.com/cdk/v2/guide/get_context_var.html) or via the flag `-c`of the cdk cli.\n\nThe context used for this override is `sops_sync_provider_asset_path`.\n\nSo for example you can use\n\n```bash\ncdk deploy -c \"sops_sync_provider_asset_path=some/path/asset.zip\"\n```\n\nor in your cdk.json\n\n```json\n{\n \"context\": {\n \"sops_sync_provider_asset_path\": \"some/path/asset.zip\"\n }\n}\n```\n\n## Motivation\n\nI have created this project to solve a recurring problem of syncing Mozilla/sops secrets into AWS SecretsManager in a convenient, secure way.\n\nOther than that, or perhaps more importantly, my goal was to learn new things:\n\n* Write a Golang lambda\n* Writing unit tests incl. mocks in Golang\n* Reproducible builds of Golang binaries (byte-by-byte identical)\n* Build reproducible zips (byte-by-byte identical)\n* Release a NPM package\n* Setting up projects with projen\n* CI/CD with GitHub actions\n* CDK unit and integration tests\n\n## Other Tools like this\n\nThe problem this Construct addresses is so good, already two other implementations exist:\n\n* [isotoma/sops-secretsmanager-cdk](https://github.com/isotoma/sops-secretsmanager-cdk): Does nearly the same. Uses CustomResource, wraps the sops CLI, does not support flatten. Found it after I published my solution to NPM :-/\n* [taimos/secretsmanager-versioning](https://github.com/taimos/secretsmanager-versioning): Different approach on the same problem. This is a CLI tool with very nice integration into CDK and also handles git versioning information.\n\n## License\n\nThe Apache-2.0 license. Please have a look at the [LICENSE](LICENSE) and [LICENSE-3RD-PARTY](LICENSE-3RD-PARTY).\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "CDK Constructs that syncs your sops secrets into AWS SecretsManager secrets.",
"version": "1.6.16",
"project_urls": {
"Homepage": "https://constructs.dev/packages/cdk-sops-secrets",
"Source": "https://github.com/dbsystel/cdk-sops-secrets.git"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "da17113d9fa9c459798272b9d0bda35402a2d9932c97fd6bfc158c79bfb1efd6",
"md5": "40af198ab25a2ab6bc9c030b28a2f407",
"sha256": "0c3a05e67d91291e5b908da7ad1cde8132c5757b2a4e8bea2c79a276a17f30bd"
},
"downloads": -1,
"filename": "cdk_sops_secrets-1.6.16-py3-none-any.whl",
"has_sig": false,
"md5_digest": "40af198ab25a2ab6bc9c030b28a2f407",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "~=3.8",
"size": 8989633,
"upload_time": "2024-03-24T21:25:09",
"upload_time_iso_8601": "2024-03-24T21:25:09.853907Z",
"url": "https://files.pythonhosted.org/packages/da/17/113d9fa9c459798272b9d0bda35402a2d9932c97fd6bfc158c79bfb1efd6/cdk_sops_secrets-1.6.16-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "48ed37f2d2b8da047edc63ffdf4c0da195aaf8cee32448925497cdd394293054",
"md5": "3591cd90b2699b051a38067f16a32cb4",
"sha256": "db16a809a7b23fa775d5300fc0dcfd71fab714a94e926c6a1ef42a8d311cce73"
},
"downloads": -1,
"filename": "cdk-sops-secrets-1.6.16.tar.gz",
"has_sig": false,
"md5_digest": "3591cd90b2699b051a38067f16a32cb4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "~=3.8",
"size": 8991327,
"upload_time": "2024-03-24T21:25:17",
"upload_time_iso_8601": "2024-03-24T21:25:17.484653Z",
"url": "https://files.pythonhosted.org/packages/48/ed/37f2d2b8da047edc63ffdf4c0da195aaf8cee32448925497cdd394293054/cdk-sops-secrets-1.6.16.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-24 21:25:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "dbsystel",
"github_project": "cdk-sops-secrets",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "cdk-sops-secrets"
}
JSON
