# tg-wrap
This app simply wraps terragrunt (which is a wrapper around terraform, which is a wrapper around cloud APIs, which is...).
Wait, why on earth do we need a wrapper for a wrapper (for a wrapper)?
Well, first of all it is pretty opinionated so what works for us, doesn't necessarily work for you.
But our reasoning for creating this is as follows:
## 1. Less typing
terraform is great, and in combination with terragrunt even greater! But let's face it, terragrunt does not excel in conciseness! The options are pretty long, which leads to lots of typing. We don't like typing!
## 2. Testing modules locally
However, more importantly, we are heavily utilising [TERRAGRUNT_SOURCE](https://terragrunt.gruntwork.io/docs/features/execute-terraform-commands-on-multiple-modules-at-once/#testing-multiple-modules-locally) when developing.
The thing is that as long as you use `run-all` you can use one setting for that variable (and conveniently set it as an environment variable), while if you run a regular command, you need to specify the full path. Which is obviously different for each project.
Which leads to (even) more typing, and worse: a higher chance for errors.
Luckily you can use `run-all` and add the appriopriate flags to ensure it behaves like a regular plan|apply|destroy etc. But again, more typing.
Nothing a [bunch a aliases](https://gitlab.com/lunadata/terragrunt-utils/-/blob/main/tg-shell.sh) can't solve though!
## 3. But the original reason was: Errors when using run-all are challenging
One of the main boons of terragrunt is the ability to break up large projects in smaller steps while still retaining the inter-dependencies. However, when working on such a large project and something goes wrong somewhere in the middle is pretty challenging.
terragrunt's error messages are pretty massive, and this is extrapolated with every individual project in your dependency chain.
And if it fails somewhere at the front, it keeps on trying until the last one, blowing up your terminal in the process.
So we wanted a possibility to run the projects step by step, using the dependency graph of terragrunt and have a bit more control over it.
And when you can run it step by step, you can make the process also re-startable, which is also pretty handy!
And this was not something a bunch of aliases could solve, hence this wrapper was born. And while we we're at it, replacing the aliases with this was then pretty straightforward next step as well.
## 4. Analyzing plan files
When using the run-all, analyzing what is about to be changed is not going to be easier. Hence we created the `tgwrap analyze` function that lists all the planned changes and (if a config file is availabe) calculates a drift score and runs a [terrasafe](https://pypi.org/project/terrasafe/) style validation check.
> you can ignore minor changes, such as tag updates, with `tgwrap analyze -i tags`
It needs a config file as follows:
```yaml
---
#
# Critically of resources as interpreted by 'tgwrap analyze'.
# It uses it for a 'terrasafe' like validation if resources can safely be deleted.
# On top of that it tries to analyze and quantify the drift impact of the changes,
# so that this can be monitored.
#
low:
# defaults:
# terrasafe_level: ignore_deletions
# drift_impact:
# default: minor
# delete: medium
azuread_application.: {} # if we you want to use the defaults
azuread_app_role_assignment: # or if you want to override these
drift_impact:
delete: minor
# and so on, and so forth
medium:
# defaults:
# terrasafe_level: ignore_deletion_if_recreation
# drift_impact:
# default: medium
# delete: major
azurerm_data_factory_linked_service_key_vault.: {}
# and so on, and so forth
high:
# defaults:
# terrasafe_level: unauthorized_deletion
# drift_impact:
# default: major
# update: medium
azuread_group.:
drift_impact:
create: minor
update: minor
azurerm_application_insights.: {}
# and so on, and so forth
```
### Speeding up the performance of analyze
This `analyze` function turned out to be pretty slow, where most of the time went into the `terragrunt show` function that is executed for each individual module.
This was a bit surprising as the plan file is already available on the file system, but it turns out that terragrunt is taking quite a bit of time for managing the depdencies. Even when you're excluding the external dependencies and are located in a particular module.
So, if you add the following to your root `terragrunt.hcl`:
```hcl
terraform {
after_hook "link_to_current_module" {
commands = ["init", "plan", "apply", "validate", "destroy"]
execute = ["bash", "-c", "ln -sf $(pwd) ${get_terragrunt_dir()}/.terragrunt-cache/current"]
}
}
```
The directory where the plan file is stored (including the other resources that terraform needs) becomes predictable and it becomes possible to run a native `terraform show` (instead `terragrunt show`) which dramatically speed up things.
Just set the proper value as an environment variable:
```console
export TGWRAP_PLANFILE_DIR=".terragrunt-cache/current"
```
Or pass it along with the `--planfile-dir|-P` option and it will use that.
### Logging the results
`tgwrap` supports logging the analyze results to an [Azure Log Analytics](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-analytics-overview) custom table.
For that, the custom table need to be present, including a [data collection endpoint](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-endpoint-overview?tabs=portal) and associated [data collection rule](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-overview?tabs=portal).
When you want to activate this, just pass `--data-collection-endpoint` (or, more conveniently, set the `TGWRAP_ANALYZE_DATA_COLLECTION_ENDPOINT` environment variable) with the url to which the data can be posted.
> Note that for this to work, `tgwrap` assumes that there is a functioning [azure cli](https://learn.microsoft.com/en-us/cli/azure/) available on the system.
A payload as below will be posted, and the log analytics table should be able to accomodate for that:
```json
[
{
"scope": "terragrunt/dlzs/data-platform/global/platform/rbac/",
"principal": "myself",
"repo": "https://gitlab.com/my-git-repo.git",
"creations": 0,
"updates": 0,
"deletions": 0,
"minor": 0,
"medium": 0,
"major": 0,
"unknown": 0,
"total": 0,
"score": 0.0,
"details": [
{
"drifts": {
"minor": 0,
"medium": 0,
"major": 0,
"unknown": 0,
"total": 0,
"score": 0.0
},
"all": [],
"creations": [],
"updates": [],
"deletions": [],
"unauthorized": [],
"unknowns": [],
"module": ""
}
]
}
]
```
The log analytics (custom) table should have a schema that is able to cope with the message above:
| Field | Type |
|----------------|----------|
| creations | Int |
| deletions | Int |
| details | Dynamic |
| major | Int |
| medium | Int |
| minor | Int |
| principal | String |
| repo | String |
| scope | String |
| score | Int |
| TimeGenerated | Datetime |
| total | Int |
| unknown | Int |
| updates | Int |
## More than a wrapper
Over time, tgwrap became more than a wrapper, blantly violating [#1 of the unix philosophy](https://en.wikipedia.org/wiki/Unix_philosophy#:~:text=The%20Unix%20philosophy%20is%20documented,%2C%20as%20yet%20unknown%2C%20program.): 'Make each program do one thing well'.
For instance, the 'analyze' functionality is already an example, but more features such as deploying a landing zone has crept into the application. It makes sense for how we're using it, but we're fully aware this makes it less generically applicable.
## Usage
```console
# general help
tgwrap --help
tgwrap run -h
tgwrap run-all -h
# run a plan
tgwrap plan # which is the same as tgwrap run plan
# run-all a plan
tgwrap run-all plan
# run-all with excluding a particular directory
tgwrap run-all plan -E 'excluded-dir/*'
# or a directory somewhere further down in the path
tgwrap run-all plan -E '**/excluded-dir/**'
# or do the same in step-by-step mode
tgwrap run-all plan -s
# or excluding (aka ignoring) external dependencies
tgwrap run-all plan -sx
# if you want to add additional arguments it is recommended to use -- as separator (although it *might* work without)
tgwrap output -- -json
```
> Note: special precautions are needed when passing on parameters that contain quotes. For instance, if you want to move state like below, escape the double quote in the staate address:
`tgwrap state mv 'azuread_group.this[\"viewers\"]' 'azuread_group.this[\"readers\"]'`
## A word about escaping inputs
Your shell is escaping special characters such as `*` and `"` before passing it to the program (`tgwrap` in this case). So some inputs **need** to be escaped in order to function properly.
For example:
```console
# to exclude certain modules from an action (such as analyze):
tgwrap analyze -E 'integrations/\*/\*'
# to import a resource that has a " in its address:
tgwrap import -a 'azuread_group.this[\"my_group\"]' -i ${GROUP_ID}
```
## Deploy manifests
In order to easily deploy a new version of the terraform (and associated terragrunt) modules, we include a manifest file in the root of the landing zone:
```yaml
---
git_repository: ssh://git@gitlab.com/my-org/my-terraform-modules-repo.git
base_path: terragrunt/my-platform # path where the terragrunt modules are stored
config_path: terragrunt/config/platform-dev # path (relative to base path) where the config files are stored
deploy: # which modules do you want to deploy
dtap:
applies_to_stages:
- dev
- tst
- acc
- prd
source_stage: dev
source_dir: platform # optional, if the source modules are not directly in the stage dir, but in <stage>/<source_dir> directory
base_dir: platform # optional, if you want to deploy the base modules in its own dir, side by side with substacks
config_dir: ../../../config # this is relative to where you run the deploy
configs:
- my-config.hcl
- ../my-ss-config-dir
# skip_all_modules: True # could be handy if you only want to deploy substacks and no regular modules
exclude_modules: # these modules will always be excluded, can be omitted
- my-specific-module
include_modules: {} # omit or use an empty dict for all of them
# or specify your modules as follows
# base: {} # just a simple include
# networking-connected: # or a bit more complicated
# - source: networking
# - target: networking-connected
substacks:
is01:
source: shared-integration/intsvc01
target: integration/is01
exclude_modules: # a list of modules that will always be excluded, can be omitted
- my-specific-module
configs:
- my-ss-config.hcl
- ../my-ss-config-dir
is02:
applies_to_stages: # optional list of stages to include the substack in
- dev
source: shared-integration/intsvc01
target: integration/is02
# global configuration files that are deployed as well
# note that these files are typically applicable to all landing zones and stages!
# this might lead to unexpected behaviour
# note that you can exclude syncing these files with a command line switch
global_config_files:
root-terragrunt:
source: ../../terragrunt.hcl # relative to base_path
target: ../../terragrunt.hcl # can be omitted, then it is same as source path
terrasafe-config:
source: ../../terrasafe-config.json
```
## Inspecting deployed infrastructure
Testing infra-as-code is hard, even though test frameworks are becoming more common these days. But the standard test approaches typically work with temporary infrastructures, while it is often also useful to test a deployed infrastructure.
Frameworks like [Chef's InSpec](https://docs.chef.io/inspec/) aims at solving that, but it is pretty config management heavy (but there are add-ons for aws and azure infra). It has a steep learning curve, we only need a tiny part of it, and also comes with a commercial license.
For what we need ('is infra deployed and are the main role assignments still in place') it was pretty easy to implement in python.
For this, you can now run the `inspect` command, which will then inspect real infrastructure and role assignments, and report back whether it meets the expectations (as declared in a config file):
```yaml
---
location:
code: westeurope
full: West Europe
# the entra id groups ar specified as a map as these will be checked for existence
# but also used for role assignment validation
entra_id_groups:
platform_admins: '{domain}-platform-admins'
cost_admins: '{domain}-cost-admins'
data_admins: '{domain}-data-admins'
just_testing: group-does-not-exist
# the resources to check
resources:
- identifier: 'kv-{domain}-euw-{stage}-base'
# due to length limitations in resource names, some shortening in the name might have taken place
# so you can provide alternative ids
alternative_ids:
- 'kv-{domain}-euw-{stage}-bs'
- 'kv{domain}euw{stage}bs'
- 'kv{domain}euw{stage}base'
type: key_vault
resource_group: 'rg-{domain}-euw-{stage}-base'
role_assignments:
- platform_admins: Owner
- platform_admins: Key Vault Secrets Officer
- data_admins: Key Vault Secrets Officer
```
After which you can run the following:
```console
tgwrap inspect -d domain -s sbx -a 886d4e58-a178-4c50-ae65-xxxxxxxxxx -c ./inspect-config.yml
......
Inspection status:
entra_id_group: dps-platform-admins
-> Resource: OK (Resource dps-platform-admins of type entra_id_group OK)
entra_id_group: dps-cost-admins
-> Resource: OK (Resource dps-cost-admins of type entra_id_group OK)
entra_id_group: dps-data-admins
-> Resource: OK (Resource dps-data-admins of type entra_id_group OK)
entra_id_group: group-does-not-exist
-> Resource: NEX (Resource group-does-not-exist of type entra_id_group not found)
key_vault: kv-dps-euw-sbx-base
-> Resource: OK (Resource kv-dps-euw-sbx-base of type key_vault OK)
-> RBAC: NOK (Principal platform_admins has NOT role Owner assigned; )
subscription: 886d4e58-a178-4c50-ae65-xxxxxxxxxx
-> Resource: OK (Resource 886d4e58-a178-4c50-ae65-xxxxxxxxxx of type subscription OK)
-> RBAC: NC (Role assignments not checked)
```
You can sent the results also to a data collection endpoint (seel also [Logging the results](#logging-the-results)).
For that, a custom table should exist with the following structure:
| Field | Type |
|----------------------------|----------|
| domain | String |
| substack | String |
| stage | String |
| subscription_id | String |
| resource_type | String |
| inspect_status_code | String |
| inspect_status | String |
| inspect_message | String |
| rbac_assignment_status_code| String |
| rbac_assignment_status | String |
| rbac_assignment_message | String |
| resource | String |
## Generating change logs
tgwrap can generate a change log by running:
```console
tgwrap change-log [--changelog-file ./CHANGELOG.md]
```
The (optional) change log file will be, if passed along. tgwrap then checks the file for the existance of a start and end markers, in the following format:
```python
start_marker = '<!-- BEGINNING OF OF TGWRAP CHANGELOG SECTION -->'
end_marker = '<!-- END OF TGWRAP CHANGELOG SECTION -->'
```
If they exist, everything between these lines will be replaced by the new change log.
## Development
In order to develop, you need to apply it to your terragrunt projects. For that you can use the `--terragrunt-working-dir` option and just run it from the poetry directory. Alternatively you can use the [tgwrap-dev](./tgwrap-dev) script and invoke that from your terragrunt directories. Either put it in your `PATH` or create an alias for convenience.
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/lunadata/tgwrap",
"name": "tgwrap",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.12",
"maintainer_email": null,
"keywords": "terraform, terragrunt, terrasafe, python",
"author": "Gerco Grandia",
"author_email": "gerco.grandia@4synergy.nl",
"download_url": "https://files.pythonhosted.org/packages/90/a0/406f37e111936df657a4194aa0ff117e52372fd4728aa297a38a6d3d5352/tgwrap-0.11.0.tar.gz",
"platform": null,
"description": "# tg-wrap\n\nThis app simply wraps terragrunt (which is a wrapper around terraform, which is a wrapper around cloud APIs, which is...).\n\nWait, why on earth do we need a wrapper for a wrapper (for a wrapper)?\n\nWell, first of all it is pretty opinionated so what works for us, doesn't necessarily work for you.\n\nBut our reasoning for creating this is as follows:\n\n## 1. Less typing\n\nterraform is great, and in combination with terragrunt even greater! But let's face it, terragrunt does not excel in conciseness! The options are pretty long, which leads to lots of typing. We don't like typing!\n\n## 2. Testing modules locally\n\nHowever, more importantly, we are heavily utilising [TERRAGRUNT_SOURCE](https://terragrunt.gruntwork.io/docs/features/execute-terraform-commands-on-multiple-modules-at-once/#testing-multiple-modules-locally) when developing.\n\nThe thing is that as long as you use `run-all` you can use one setting for that variable (and conveniently set it as an environment variable), while if you run a regular command, you need to specify the full path. Which is obviously different for each project.\n\nWhich leads to (even) more typing, and worse: a higher chance for errors.\n\nLuckily you can use `run-all` and add the appriopriate flags to ensure it behaves like a regular plan|apply|destroy etc. But again, more typing.\n\nNothing a [bunch a aliases](https://gitlab.com/lunadata/terragrunt-utils/-/blob/main/tg-shell.sh) can't solve though!\n\n## 3. But the original reason was: Errors when using run-all are challenging\n\nOne of the main boons of terragrunt is the ability to break up large projects in smaller steps while still retaining the inter-dependencies. However, when working on such a large project and something goes wrong somewhere in the middle is pretty challenging.\n\nterragrunt's error messages are pretty massive, and this is extrapolated with every individual project in your dependency chain.\n\nAnd if it fails somewhere at the front, it keeps on trying until the last one, blowing up your terminal in the process.\n\nSo we wanted a possibility to run the projects step by step, using the dependency graph of terragrunt and have a bit more control over it.\n\nAnd when you can run it step by step, you can make the process also re-startable, which is also pretty handy!\n\nAnd this was not something a bunch of aliases could solve, hence this wrapper was born. And while we we're at it, replacing the aliases with this was then pretty straightforward next step as well.\n\n## 4. Analyzing plan files\n\nWhen using the run-all, analyzing what is about to be changed is not going to be easier. Hence we created the `tgwrap analyze` function that lists all the planned changes and (if a config file is availabe) calculates a drift score and runs a [terrasafe](https://pypi.org/project/terrasafe/) style validation check.\n\n> you can ignore minor changes, such as tag updates, with `tgwrap analyze -i tags`\n\nIt needs a config file as follows:\n\n```yaml\n---\n#\n# Critically of resources as interpreted by 'tgwrap analyze'.\n# It uses it for a 'terrasafe' like validation if resources can safely be deleted.\n# On top of that it tries to analyze and quantify the drift impact of the changes,\n# so that this can be monitored.\n#\nlow:\n # defaults:\n # terrasafe_level: ignore_deletions\n # drift_impact:\n # default: minor\n # delete: medium\n azuread_application.: {} # if we you want to use the defaults\n azuread_app_role_assignment: # or if you want to override these\n drift_impact:\n delete: minor\n # and so on, and so forth\nmedium:\n # defaults:\n # terrasafe_level: ignore_deletion_if_recreation\n # drift_impact:\n # default: medium\n # delete: major\n azurerm_data_factory_linked_service_key_vault.: {}\n # and so on, and so forth\nhigh:\n # defaults:\n # terrasafe_level: unauthorized_deletion\n # drift_impact:\n # default: major\n # update: medium\n azuread_group.:\n drift_impact:\n create: minor\n update: minor\n azurerm_application_insights.: {}\n # and so on, and so forth\n```\n\n### Speeding up the performance of analyze\n\nThis `analyze` function turned out to be pretty slow, where most of the time went into the `terragrunt show` function that is executed for each individual module. \n\nThis was a bit surprising as the plan file is already available on the file system, but it turns out that terragrunt is taking quite a bit of time for managing the depdencies. Even when you're excluding the external dependencies and are located in a particular module.\n\nSo, if you add the following to your root `terragrunt.hcl`:\n\n```hcl\nterraform {\n after_hook \"link_to_current_module\" {\n commands = [\"init\", \"plan\", \"apply\", \"validate\", \"destroy\"]\n execute = [\"bash\", \"-c\", \"ln -sf $(pwd) ${get_terragrunt_dir()}/.terragrunt-cache/current\"]\n }\n}\n```\n\nThe directory where the plan file is stored (including the other resources that terraform needs) becomes predictable and it becomes possible to run a native `terraform show` (instead `terragrunt show`) which dramatically speed up things.\n\nJust set the proper value as an environment variable:\n\n```console\nexport TGWRAP_PLANFILE_DIR=\".terragrunt-cache/current\"\n```\n\nOr pass it along with the `--planfile-dir|-P` option and it will use that.\n\n### Logging the results\n\n`tgwrap` supports logging the analyze results to an [Azure Log Analytics](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-analytics-overview) custom table.\n\nFor that, the custom table need to be present, including a [data collection endpoint](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-endpoint-overview?tabs=portal) and associated [data collection rule](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-overview?tabs=portal).\n\nWhen you want to activate this, just pass `--data-collection-endpoint` (or, more conveniently, set the `TGWRAP_ANALYZE_DATA_COLLECTION_ENDPOINT` environment variable) with the url to which the data can be posted.\n\n> Note that for this to work, `tgwrap` assumes that there is a functioning [azure cli](https://learn.microsoft.com/en-us/cli/azure/) available on the system.\n\nA payload as below will be posted, and the log analytics table should be able to accomodate for that:\n\n```json\n[\n {\n \"scope\": \"terragrunt/dlzs/data-platform/global/platform/rbac/\",\n \"principal\": \"myself\",\n \"repo\": \"https://gitlab.com/my-git-repo.git\",\n \"creations\": 0,\n \"updates\": 0,\n \"deletions\": 0,\n \"minor\": 0,\n \"medium\": 0,\n \"major\": 0,\n \"unknown\": 0,\n \"total\": 0,\n \"score\": 0.0,\n \"details\": [\n {\n \"drifts\": {\n \"minor\": 0,\n \"medium\": 0,\n \"major\": 0,\n \"unknown\": 0,\n \"total\": 0,\n \"score\": 0.0\n },\n \"all\": [],\n \"creations\": [],\n \"updates\": [],\n \"deletions\": [],\n \"unauthorized\": [],\n \"unknowns\": [],\n \"module\": \"\"\n }\n ]\n }\n]\n```\n\nThe log analytics (custom) table should have a schema that is able to cope with the message above:\n\n| Field | Type |\n|----------------|----------|\n| creations | Int |\n| deletions | Int |\n| details | Dynamic |\n| major | Int |\n| medium | Int |\n| minor | Int |\n| principal | String |\n| repo | String |\n| scope | String |\n| score | Int |\n| TimeGenerated | Datetime |\n| total | Int |\n| unknown | Int |\n| updates | Int |\n\n## More than a wrapper\n\nOver time, tgwrap became more than a wrapper, blantly violating [#1 of the unix philosophy](https://en.wikipedia.org/wiki/Unix_philosophy#:~:text=The%20Unix%20philosophy%20is%20documented,%2C%20as%20yet%20unknown%2C%20program.): 'Make each program do one thing well'.\n\nFor instance, the 'analyze' functionality is already an example, but more features such as deploying a landing zone has crept into the application. It makes sense for how we're using it, but we're fully aware this makes it less generically applicable.\n\n## Usage\n\n```console\n# general help\ntgwrap --help\n\ntgwrap run -h\ntgwrap run-all -h\n\n# run a plan\ntgwrap plan # which is the same as tgwrap run plan\n\n# run-all a plan\ntgwrap run-all plan\n\n# run-all with excluding a particular directory\ntgwrap run-all plan -E 'excluded-dir/*'\n# or a directory somewhere further down in the path\ntgwrap run-all plan -E '**/excluded-dir/**'\n\n# or do the same in step-by-step mode\ntgwrap run-all plan -s\n\n# or excluding (aka ignoring) external dependencies\ntgwrap run-all plan -sx\n\n# if you want to add additional arguments it is recommended to use -- as separator (although it *might* work without)\ntgwrap output -- -json\n```\n\n> Note: special precautions are needed when passing on parameters that contain quotes. For instance, if you want to move state like below, escape the double quote in the staate address:\n\n`tgwrap state mv 'azuread_group.this[\\\"viewers\\\"]' 'azuread_group.this[\\\"readers\\\"]'`\n\n## A word about escaping inputs\n\nYour shell is escaping special characters such as `*` and `\"` before passing it to the program (`tgwrap` in this case). So some inputs **need** to be escaped in order to function properly.\n\nFor example:\n\n```console\n# to exclude certain modules from an action (such as analyze):\ntgwrap analyze -E 'integrations/\\*/\\*'\n\n# to import a resource that has a \" in its address:\ntgwrap import -a 'azuread_group.this[\\\"my_group\\\"]' -i ${GROUP_ID}\n```\n\n## Deploy manifests\n\nIn order to easily deploy a new version of the terraform (and associated terragrunt) modules, we include a manifest file in the root of the landing zone:\n\n```yaml\n---\ngit_repository: ssh://git@gitlab.com/my-org/my-terraform-modules-repo.git\nbase_path: terragrunt/my-platform # path where the terragrunt modules are stored \nconfig_path: terragrunt/config/platform-dev # path (relative to base path) where the config files are stored\n\ndeploy: # which modules do you want to deploy\n dtap:\n applies_to_stages:\n - dev\n - tst\n - acc\n - prd\n source_stage: dev\n source_dir: platform # optional, if the source modules are not directly in the stage dir, but in <stage>/<source_dir> directory\n base_dir: platform # optional, if you want to deploy the base modules in its own dir, side by side with substacks\n config_dir: ../../../config # this is relative to where you run the deploy\n configs:\n - my-config.hcl\n - ../my-ss-config-dir\n # skip_all_modules: True # could be handy if you only want to deploy substacks and no regular modules\n exclude_modules: # these modules will always be excluded, can be omitted\n - my-specific-module\n include_modules: {} # omit or use an empty dict for all of them\n # or specify your modules as follows\n # base: {} # just a simple include\n # networking-connected: # or a bit more complicated\n # - source: networking\n # - target: networking-connected\n\nsubstacks:\n is01:\n source: shared-integration/intsvc01\n target: integration/is01\n exclude_modules: # a list of modules that will always be excluded, can be omitted\n - my-specific-module\n configs:\n - my-ss-config.hcl\n - ../my-ss-config-dir\n is02:\n applies_to_stages: # optional list of stages to include the substack in\n - dev\n source: shared-integration/intsvc01\n target: integration/is02\n\n# global configuration files that are deployed as well\n# note that these files are typically applicable to all landing zones and stages!\n# this might lead to unexpected behaviour\n# note that you can exclude syncing these files with a command line switch\nglobal_config_files:\n root-terragrunt:\n source: ../../terragrunt.hcl # relative to base_path\n target: ../../terragrunt.hcl # can be omitted, then it is same as source path\n terrasafe-config:\n source: ../../terrasafe-config.json\n```\n\n## Inspecting deployed infrastructure\n\nTesting infra-as-code is hard, even though test frameworks are becoming more common these days. But the standard test approaches typically work with temporary infrastructures, while it is often also useful to test a deployed infrastructure.\n\nFrameworks like [Chef's InSpec](https://docs.chef.io/inspec/) aims at solving that, but it is pretty config management heavy (but there are add-ons for aws and azure infra). It has a steep learning curve, we only need a tiny part of it, and also comes with a commercial license.\n\nFor what we need ('is infra deployed and are the main role assignments still in place') it was pretty easy to implement in python.\n\nFor this, you can now run the `inspect` command, which will then inspect real infrastructure and role assignments, and report back whether it meets the expectations (as declared in a config file):\n\n```yaml\n---\nlocation:\n code: westeurope\n full: West Europe\n\n# the entra id groups ar specified as a map as these will be checked for existence\n# but also used for role assignment validation\nentra_id_groups:\n platform_admins: '{domain}-platform-admins'\n cost_admins: '{domain}-cost-admins'\n data_admins: '{domain}-data-admins'\n just_testing: group-does-not-exist\n\n# the resources to check\nresources:\n - identifier: 'kv-{domain}-euw-{stage}-base'\n # due to length limitations in resource names, some shortening in the name might have taken place\n # so you can provide alternative ids\n alternative_ids:\n - 'kv-{domain}-euw-{stage}-bs'\n - 'kv{domain}euw{stage}bs'\n - 'kv{domain}euw{stage}base'\n type: key_vault\n resource_group: 'rg-{domain}-euw-{stage}-base'\n role_assignments:\n - platform_admins: Owner\n - platform_admins: Key Vault Secrets Officer\n - data_admins: Key Vault Secrets Officer\n```\n\nAfter which you can run the following:\n\n```console\ntgwrap inspect -d domain -s sbx -a 886d4e58-a178-4c50-ae65-xxxxxxxxxx -c ./inspect-config.yml\n......\n\nInspection status:\nentra_id_group: dps-platform-admins\n\t-> Resource:\tOK (Resource dps-platform-admins of type entra_id_group OK)\nentra_id_group: dps-cost-admins\n\t-> Resource:\tOK (Resource dps-cost-admins of type entra_id_group OK)\nentra_id_group: dps-data-admins\n\t-> Resource:\tOK (Resource dps-data-admins of type entra_id_group OK)\nentra_id_group: group-does-not-exist\n\t-> Resource:\tNEX (Resource group-does-not-exist of type entra_id_group not found)\nkey_vault: kv-dps-euw-sbx-base\n\t-> Resource:\tOK (Resource kv-dps-euw-sbx-base of type key_vault OK)\n\t-> RBAC:\tNOK (Principal platform_admins has NOT role Owner assigned; )\nsubscription: 886d4e58-a178-4c50-ae65-xxxxxxxxxx\n\t-> Resource:\tOK (Resource 886d4e58-a178-4c50-ae65-xxxxxxxxxx of type subscription OK)\n\t-> RBAC:\tNC (Role assignments not checked)\n```\n\nYou can sent the results also to a data collection endpoint (seel also [Logging the results](#logging-the-results)).\n\nFor that, a custom table should exist with the following structure:\n\n\n| Field | Type |\n|----------------------------|----------|\n| domain | String |\n| substack | String |\n| stage | String |\n| subscription_id | String |\n| resource_type | String |\n| inspect_status_code | String |\n| inspect_status | String |\n| inspect_message | String |\n| rbac_assignment_status_code| String |\n| rbac_assignment_status | String |\n| rbac_assignment_message | String |\n| resource | String |\n\n## Generating change logs\n\ntgwrap can generate a change log by running:\n\n```console\ntgwrap change-log [--changelog-file ./CHANGELOG.md]\n```\n\nThe (optional) change log file will be, if passed along. tgwrap then checks the file for the existance of a start and end markers, in the following format:\n\n```python\n start_marker = '<!-- BEGINNING OF OF TGWRAP CHANGELOG SECTION -->'\n end_marker = '<!-- END OF TGWRAP CHANGELOG SECTION -->'\n```\n\nIf they exist, everything between these lines will be replaced by the new change log.\n\n## Development\n\nIn order to develop, you need to apply it to your terragrunt projects. For that you can use the `--terragrunt-working-dir` option and just run it from the poetry directory. Alternatively you can use the [tgwrap-dev](./tgwrap-dev) script and invoke that from your terragrunt directories. Either put it in your `PATH` or create an alias for convenience.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A (terragrunt) wrapper around a (terraform) wrapper around ....",
"version": "0.11.0",
"project_urls": {
"Documentation": "https://gitlab.com/lunadata/tgwrap/",
"Homepage": "https://gitlab.com/lunadata/tgwrap",
"Repository": "https://gitlab.com/lunadata/tgwrap"
},
"split_keywords": [
"terraform",
" terragrunt",
" terrasafe",
" python"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e10684ff4bc24eb60126a8d13f2a1ff140ca8e9fa48e730395ada50d6487779b",
"md5": "f2dfb19042858f9fb2cbed03d38aaf7b",
"sha256": "97ae663806c038c40f06cecdb75daf408e26d02c49364d1dcd27e843d5acda9d"
},
"downloads": -1,
"filename": "tgwrap-0.11.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f2dfb19042858f9fb2cbed03d38aaf7b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.12",
"size": 46691,
"upload_time": "2024-10-30T13:58:47",
"upload_time_iso_8601": "2024-10-30T13:58:47.441784Z",
"url": "https://files.pythonhosted.org/packages/e1/06/84ff4bc24eb60126a8d13f2a1ff140ca8e9fa48e730395ada50d6487779b/tgwrap-0.11.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "90a0406f37e111936df657a4194aa0ff117e52372fd4728aa297a38a6d3d5352",
"md5": "4772f78923aa635afd58f5c3dc83dc2b",
"sha256": "83ff910e9bda8575830903f06d501e6c7725864f74ba28ff41d9da5eec23af75"
},
"downloads": -1,
"filename": "tgwrap-0.11.0.tar.gz",
"has_sig": false,
"md5_digest": "4772f78923aa635afd58f5c3dc83dc2b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.12",
"size": 49696,
"upload_time": "2024-10-30T13:58:49",
"upload_time_iso_8601": "2024-10-30T13:58:49.350562Z",
"url": "https://files.pythonhosted.org/packages/90/a0/406f37e111936df657a4194aa0ff117e52372fd4728aa297a38a6d3d5352/tgwrap-0.11.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-30 13:58:49",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "lunadata",
"gitlab_project": "tgwrap",
"lcname": "tgwrap"
}
JSON