[](https://pypi.org/project/cloud-governance/)
[](https://quay.io/repository/cloud-governance/cloud-governance?tab=tags)
[](https://github.com/redhat-performance/cloud-governance/actions)[](https://coveralls.io/github/redhat-performance/cloud-governance?branch=main)
[](https://cloud-governance.readthedocs.io/en/latest/?badge=latest)
[](https://pypi.org/project/cloud-governance)
[](https://github.com/redhat-performance/cloud-governance/blob/main/LICENSE)
# Cloud Governance
## What is it?
**Cloud Governance** tool provides a lightweight and flexible framework for deploying cloud management policies focusing
on cost optimize and security.
We have implemented several pruning policies. \
When monitoring the resources, we found that most of the cost leakage is from available volumes, unused NAT gateways,
and unattached Public IPv4 addresses (Starting from February 2024, public IPv4 addresses are chargeable whether they are
used or not).
| Providers | Disks | NatGateway | PublicIp | Snapshots | InstanceIdle | TagResources | EC2Stop | ocp_cleanup | ClusterRun | EmptyBucket | EmptyRoles |
|-----------|---------|------------|----------|-----------|--------------|--------------|---------|-------------|------------|-------------|------------|
| AWS | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Azure | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
List of Policies:
##### [AWS Polices!](./POLICIES.md#aws-policies)
- instance_idle
- instance_run
- unattached_volume
- zombie_cluster_resource
- ip_unattached
- zombie_snapshots
- unused_nat_gateway
- s3_inactive
- empty_roles
- tag_resources
- tag_iam_user
- cost_over_usage
- cluster_run
##### [Azure Polices!](POLICIES.md)
- instance_idle
- unattached_volume
- ip_unattached
- unused_nat_gateway
##### [IBM Polices!](POLICIES.md)
- tag_baremetal
- tag_vm
- tag_resources
Check out policy summary [here!](POLICIES.md)
Reference:
* Checkout
blog: [Optimizing cloud resource management with cloud governance](https://www.redhat.com/en/blog/optimizing-cloud-resource-management-cloud-governance)
* The cloud-governance package is placed in [PyPi](https://pypi.org/project/cloud-governance/)
* The cloud-governance container image is placed in [Quay.io](https://quay.io/repository/ebattat/cloud-governance)
* The cloud-governance readthedocs link is [ReadTheDocs](https://cloud-governance.readthedocs.io/en/latest/)
[//]: # ( )
_**Table of Contents**_
<!-- TOC -->
- [Installation](#installation)
- [Configuration](#environment-variables-configurations)
- [Run Policies](#run-policies)
- [Run Policy Using Pod](#run-policy-using-pod)
- [Pytest](#pytest)
- [Post Installation](#post-installation)
<!-- /TOC -->
## Installation
#### Download cloud-governance image from quay.io
```sh
podman pull quay.io/cloud-governance/cloud-governance
```
#### Environment variables configurations:
| Key | Value | Description |
|--------------------------------|----------|:----------------------------------------------------------------------------|
| AWS_ACCESS_KEY_ID | required | AWS access key |
| AWS_SECRET_ACCESS_KEY | required | AWS Secret key |
| AWS_DEFAULT_REGION | required | AWS Region, default set to us-east-2 |
| BUCKET_NAME | optional | Cloud bucket Name, to store data |
| policy | required | check [here](POLICIES.md) for policies list |
| dry_run | optional | default set to "yes", supported only two: yes/ no |
| log_level | optional | default set to INFO |
| LDAP_HOST_NAME | optional | ldap hostnames |
| es_host | optional | Elasticsearch Host |
| es_port | optional | Elasticsearch Port |
| es_index | optional | Elasticsearch Index, to push the data. default to cloud-governance-es-index |
| GOOGLE_APPLICATION_CREDENTIALS | optional | GCP creds, to access google resources. i.e Sheets, Docs |
| AZURE_CLIENT_SECRET | required | Azure Client Secret |
| AZURE_TENANT_ID | | Azure Tenant Id |
| AZURE_ACCOUNT_ID | | Azure Account Id |
| AZURE_CLIENT_ID | | Azure Client Id |
| GCP_DATABASE_NAME | | GCP BigQuery database name, used to generate cost reports |
| GCP_DATABASE_TABLE_NAME | | GCP BigQuery TableName, used to generate cost reports |
| IBM_API_USERNAME | | IBM Account Username |
| IBM_API_KEY | | IBM Account Classic Infrastructure key |
| IBM_CLOUD_API_KEY | | IBM Cloud API Key |
| IBM_CUSTOM_TAGS_LIST | | pass string with separated with comma. i.e: "cost-center: test, env: test" |
### AWS Configuration
Create IAM User with Read/Delete Permissions and create S3 bucket.
- Follow the instructions [README.md](iam/clouds/aws/CloudGovernanceInfra/README.md).
### IBM Configuration
* Create classic infrastructure API key
* Create IBM CLOUD API key to use tag_resources policy
## Run Policies
## AWS
- Passing environment variables
```shell
podman run --rm --name cloud-governance \
-e policy="zombie_cluster_resource" \
-e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \
-e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \
-e AWS_DEFAULT_REGION="us-east-2" \
-e dry_run="yes" \
"quay.io/cloud-governance/cloud-governance"
```
- Using involvement file config
- Create env.yaml file, and mount it to /tmp/env.yaml else mount to anypath and pass env DEFAULT_CONFIG_PATH where you
mounted
```yaml
AWS_ACCESS_KEY_ID: ""
AWS_SECRET_ACCESS_KEY: ""
AWS_DEFAULT_REGION: "us-east-2"
policy: "zombie_cluster_resource"
dry_run: "yes"
es_host: ""
es_port: ""
es_index: ""
```
```shell
podman run --rm --name cloud-governance \
-v "env.yaml":"/tmp/env.yaml" \
--net="host" \
"quay.io/cloud-governance/cloud-governance"
```
## Run Policy Using Pod
#### Run as a pod job via OpenShift
Job Pod: [cloud-governance.yaml](pod_yaml/cloud-governance.yaml)
Configmaps: [cloud_governance_configmap.yaml](pod_yaml/cloud_governance_configmap.yaml)
Quay.io Secret: [quayio_secret.sh](pod_yaml/quayio_secret.sh)
AWS Secret: [cloud_governance_secret.yaml](pod_yaml/cloud_governance_secret.yaml)
* Need to convert secret key to base64 [run_base64.py](pod_yaml/run_base64.py)
## Pytest
##### Cloud-governance integration tests using pytest
```sh
python3 -m venv governance
source governance/bin/activate
(governance) $ python -m pip install --upgrade pip
(governance) $ pip install coverage
(governance) $ pip install pytest
(governance) $ git clone https://github.com/redhat-performance/cloud-governance
(governance) $ cd cloud-governance
(governance) $ coverage run -m pytest
(governance) $ deactivate
rm -rf *governance*
```
## Post Installation
#### Delete cloud-governance image
```sh
sudo podman rmi quay.io/cloud-governance/cloud-governance
```
Raw data
{
"_id": null,
"home_page": "https://github.com/redhat-performance/cloud-governance",
"name": "cloud-governance",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "Red Hat",
"author_email": "ebattat@redhat.com, athiruma@redhat.com",
"download_url": "https://files.pythonhosted.org/packages/ce/c5/371513427734e99b9b965af08f337c47df9f470fe03158313334b511fd65/cloud_governance-1.1.338.tar.gz",
"platform": null,
"description": "[](https://pypi.org/project/cloud-governance/)\n[![Container Repository on Quay](https://quay.io/repository/projectquay/quay/status \"Container Repository on Quay\")](https://quay.io/repository/cloud-governance/cloud-governance?tab=tags)\n[](https://github.com/redhat-performance/cloud-governance/actions)[](https://coveralls.io/github/redhat-performance/cloud-governance?branch=main)\n[](https://cloud-governance.readthedocs.io/en/latest/?badge=latest)\n[](https://pypi.org/project/cloud-governance)\n[](https://github.com/redhat-performance/cloud-governance/blob/main/LICENSE)\n\n# Cloud Governance\n\n\n\n## What is it?\n\n**Cloud Governance** tool provides a lightweight and flexible framework for deploying cloud management policies focusing\non cost optimize and security.\nWe have implemented several pruning policies. \\\nWhen monitoring the resources, we found that most of the cost leakage is from available volumes, unused NAT gateways,\nand unattached Public IPv4 addresses (Starting from February 2024, public IPv4 addresses are chargeable whether they are\nused or not).\n\n| Providers | Disks | NatGateway | PublicIp | Snapshots | InstanceIdle | TagResources | EC2Stop | ocp_cleanup | ClusterRun | EmptyBucket | EmptyRoles |\n|-----------|---------|------------|----------|-----------|--------------|--------------|---------|-------------|------------|-------------|------------|\n| AWS | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |\n| Azure | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |\n\nList of Policies:\n\n##### [AWS Polices!](./POLICIES.md#aws-policies)\n\n- instance_idle\n- instance_run\n- unattached_volume\n- zombie_cluster_resource\n- ip_unattached\n- zombie_snapshots\n- unused_nat_gateway\n- s3_inactive\n- empty_roles\n- tag_resources\n- tag_iam_user\n- cost_over_usage\n- cluster_run\n\n##### [Azure Polices!](POLICIES.md)\n\n- instance_idle\n- unattached_volume\n- ip_unattached\n- unused_nat_gateway\n\n##### [IBM Polices!](POLICIES.md)\n\n- tag_baremetal\n- tag_vm\n- tag_resources\n\nCheck out policy summary [here!](POLICIES.md)\n\n\n\n\nReference:\n\n* Checkout\n blog: [Optimizing cloud resource management with cloud governance](https://www.redhat.com/en/blog/optimizing-cloud-resource-management-cloud-governance)\n* The cloud-governance package is placed in [PyPi](https://pypi.org/project/cloud-governance/)\n* The cloud-governance container image is placed in [Quay.io](https://quay.io/repository/ebattat/cloud-governance)\n* The cloud-governance readthedocs link is [ReadTheDocs](https://cloud-governance.readthedocs.io/en/latest/)\n\n[//]: # ( )\n\n_**Table of Contents**_\n\n<!-- TOC -->\n\n- [Installation](#installation)\n- [Configuration](#environment-variables-configurations)\n- [Run Policies](#run-policies)\n- [Run Policy Using Pod](#run-policy-using-pod)\n- [Pytest](#pytest)\n- [Post Installation](#post-installation)\n\n<!-- /TOC -->\n\n## Installation\n\n#### Download cloud-governance image from quay.io\n\n```sh\npodman pull quay.io/cloud-governance/cloud-governance\n```\n\n#### Environment variables configurations:\n\n| Key | Value | Description |\n|--------------------------------|----------|:----------------------------------------------------------------------------|\n| AWS_ACCESS_KEY_ID | required | AWS access key |\n| AWS_SECRET_ACCESS_KEY | required | AWS Secret key |\n| AWS_DEFAULT_REGION | required | AWS Region, default set to us-east-2 |\n| BUCKET_NAME | optional | Cloud bucket Name, to store data |\n| policy | required | check [here](POLICIES.md) for policies list |\n| dry_run | optional | default set to \"yes\", supported only two: yes/ no |\n| log_level | optional | default set to INFO |\n| LDAP_HOST_NAME | optional | ldap hostnames |\n| es_host | optional | Elasticsearch Host |\n| es_port | optional | Elasticsearch Port |\n| es_index | optional | Elasticsearch Index, to push the data. default to cloud-governance-es-index |\n| GOOGLE_APPLICATION_CREDENTIALS | optional | GCP creds, to access google resources. i.e Sheets, Docs |\n| AZURE_CLIENT_SECRET | required | Azure Client Secret |\n| AZURE_TENANT_ID | | Azure Tenant Id |\n| AZURE_ACCOUNT_ID | | Azure Account Id |\n| AZURE_CLIENT_ID | | Azure Client Id |\n| GCP_DATABASE_NAME | | GCP BigQuery database name, used to generate cost reports |\n| GCP_DATABASE_TABLE_NAME | | GCP BigQuery TableName, used to generate cost reports |\n| IBM_API_USERNAME | | IBM Account Username |\n| IBM_API_KEY | | IBM Account Classic Infrastructure key |\n| IBM_CLOUD_API_KEY | | IBM Cloud API Key |\n| IBM_CUSTOM_TAGS_LIST | | pass string with separated with comma. i.e: \"cost-center: test, env: test\" |\n\n### AWS Configuration\n\nCreate IAM User with Read/Delete Permissions and create S3 bucket.\n\n- Follow the instructions [README.md](iam/clouds/aws/CloudGovernanceInfra/README.md).\n\n### IBM Configuration\n\n* Create classic infrastructure API key\n* Create IBM CLOUD API key to use tag_resources policy\n\n## Run Policies\n\n## AWS\n\n- Passing environment variables\n\n```shell\n podman run --rm --name cloud-governance \\\n -e policy=\"zombie_cluster_resource\" \\\n -e AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\" \\\n -e AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\" \\\n -e AWS_DEFAULT_REGION=\"us-east-2\" \\\n -e dry_run=\"yes\" \\\n \"quay.io/cloud-governance/cloud-governance\"\n```\n\n- Using involvement file config\n- Create env.yaml file, and mount it to /tmp/env.yaml else mount to anypath and pass env DEFAULT_CONFIG_PATH where you\n mounted\n\n```yaml\nAWS_ACCESS_KEY_ID: \"\"\nAWS_SECRET_ACCESS_KEY: \"\"\nAWS_DEFAULT_REGION: \"us-east-2\"\npolicy: \"zombie_cluster_resource\"\ndry_run: \"yes\"\nes_host: \"\"\nes_port: \"\"\nes_index: \"\"\n```\n\n```shell\n podman run --rm --name cloud-governance \\\n -v \"env.yaml\":\"/tmp/env.yaml\" \\\n --net=\"host\" \\\n \"quay.io/cloud-governance/cloud-governance\"\n```\n\n## Run Policy Using Pod\n\n#### Run as a pod job via OpenShift\n\nJob Pod: [cloud-governance.yaml](pod_yaml/cloud-governance.yaml)\n\nConfigmaps: [cloud_governance_configmap.yaml](pod_yaml/cloud_governance_configmap.yaml)\n\nQuay.io Secret: [quayio_secret.sh](pod_yaml/quayio_secret.sh)\n\nAWS Secret: [cloud_governance_secret.yaml](pod_yaml/cloud_governance_secret.yaml)\n\n * Need to convert secret key to base64 [run_base64.py](pod_yaml/run_base64.py)\n\n## Pytest\n\n##### Cloud-governance integration tests using pytest\n\n```sh\npython3 -m venv governance\nsource governance/bin/activate\n(governance) $ python -m pip install --upgrade pip\n(governance) $ pip install coverage\n(governance) $ pip install pytest\n(governance) $ git clone https://github.com/redhat-performance/cloud-governance\n(governance) $ cd cloud-governance\n(governance) $ coverage run -m pytest\n(governance) $ deactivate\nrm -rf *governance*\n```\n\n## Post Installation\n\n#### Delete cloud-governance image\n\n```sh\nsudo podman rmi quay.io/cloud-governance/cloud-governance\n```\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "Cloud Governance Tool",
"version": "1.1.338",
"project_urls": {
"Homepage": "https://github.com/redhat-performance/cloud-governance"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "83f1cad9e1931d9a9f9ba53285265fb7499f621a97d27cab6c492684030f9acf",
"md5": "d0e71f382c2167537523b83e68ad2e6c",
"sha256": "dd8e1110894d8d3aad01190ef25e32625e4ecdc99e55e7289db95edaa8dea122"
},
"downloads": -1,
"filename": "cloud_governance-1.1.338-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d0e71f382c2167537523b83e68ad2e6c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 300637,
"upload_time": "2024-12-09T21:38:57",
"upload_time_iso_8601": "2024-12-09T21:38:57.611525Z",
"url": "https://files.pythonhosted.org/packages/83/f1/cad9e1931d9a9f9ba53285265fb7499f621a97d27cab6c492684030f9acf/cloud_governance-1.1.338-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "cec5371513427734e99b9b965af08f337c47df9f470fe03158313334b511fd65",
"md5": "410d1a06908d682ea49fbccf1d655b75",
"sha256": "da9b2f6aac95d2be60c2a0d7a56e94b67a0671708661f14e2ed4e0e1cb915a82"
},
"downloads": -1,
"filename": "cloud_governance-1.1.338.tar.gz",
"has_sig": false,
"md5_digest": "410d1a06908d682ea49fbccf1d655b75",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 204951,
"upload_time": "2024-12-09T21:39:00",
"upload_time_iso_8601": "2024-12-09T21:39:00.519188Z",
"url": "https://files.pythonhosted.org/packages/ce/c5/371513427734e99b9b965af08f337c47df9f470fe03158313334b511fd65/cloud_governance-1.1.338.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-09 21:39:00",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "redhat-performance",
"github_project": "cloud-governance",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [
{
"name": "aiohttp",
"specs": [
[
"==",
"3.10.11"
]
]
},
{
"name": "attrs",
"specs": [
[
"==",
"21.4.0"
]
]
},
{
"name": "azure-identity",
"specs": [
[
"==",
"1.16.1"
]
]
},
{
"name": "azure-mgmt-billing",
"specs": [
[
"==",
"6.0.0"
]
]
},
{
"name": "azure-mgmt-compute",
"specs": [
[
"==",
"30.1.0"
]
]
},
{
"name": "azure-mgmt-costmanagement",
"specs": [
[
"==",
"3.0.0"
]
]
},
{
"name": "azure-mgmt-monitor",
"specs": [
[
"==",
"6.0.2"
]
]
},
{
"name": "azure-mgmt-network",
"specs": [
[
"==",
"25.0.0"
]
]
},
{
"name": "azure-mgmt-resource",
"specs": [
[
"==",
"23.0.1"
]
]
},
{
"name": "azure-mgmt-subscription",
"specs": [
[
"==",
"3.1.1"
]
]
},
{
"name": "boto3",
"specs": [
[
"==",
"1.26.4"
]
]
},
{
"name": "botocore",
"specs": [
[
"==",
"1.29.4"
]
]
},
{
"name": "elasticsearch",
"specs": [
[
"==",
"7.13.4"
]
]
},
{
"name": "elasticsearch-dsl",
"specs": [
[
"==",
"7.4.0"
]
]
},
{
"name": "google-api-python-client",
"specs": [
[
"==",
"2.57.0"
]
]
},
{
"name": "google-auth-httplib2",
"specs": [
[
"==",
"0.1.0"
]
]
},
{
"name": "google-auth-oauthlib",
"specs": [
[
"==",
"0.5.2"
]
]
},
{
"name": "google-cloud-bigquery",
"specs": [
[
"==",
"3.5.0"
]
]
},
{
"name": "google-cloud-billing",
"specs": [
[
"==",
"1.9.1"
]
]
},
{
"name": "ibm-cloud-sdk-core",
"specs": [
[
"==",
"3.18.0"
]
]
},
{
"name": "ibm-cos-sdk",
"specs": [
[
"==",
"2.13.6"
]
]
},
{
"name": "ibm-platform-services",
"specs": [
[
"==",
"0.27.0"
]
]
},
{
"name": "ibm-schematics",
"specs": [
[
"==",
"1.1.0"
]
]
},
{
"name": "ibm-vpc",
"specs": [
[
"==",
"0.21.0"
]
]
},
{
"name": "myst-parser",
"specs": [
[
"==",
"1.0.0"
]
]
},
{
"name": "numpy",
"specs": [
[
"<=",
"1.26.4"
]
]
},
{
"name": "oauthlib",
"specs": [
[
"~=",
"3.1.1"
]
]
},
{
"name": "pandas",
"specs": []
},
{
"name": "PyAthena",
"specs": [
[
"==",
"3.0.5"
]
]
},
{
"name": "PyGitHub",
"specs": [
[
"==",
"1.55"
]
]
},
{
"name": "python-ldap",
"specs": [
[
"==",
"3.4.2"
]
]
},
{
"name": "requests",
"specs": [
[
"==",
"2.32.2"
]
]
},
{
"name": "retry",
"specs": [
[
"==",
"0.9.2"
]
]
},
{
"name": "setuptools",
"specs": []
},
{
"name": "SoftLayer",
"specs": [
[
"==",
"6.0.0"
]
]
},
{
"name": "sphinx",
"specs": [
[
"==",
"5.0.0"
]
]
},
{
"name": "sphinx-rtd-theme",
"specs": [
[
"==",
"1.0.0"
]
]
},
{
"name": "typeguard",
"specs": [
[
"==",
"2.13.3"
]
]
},
{
"name": "typing",
"specs": [
[
"==",
"3.7.4.3"
]
]
},
{
"name": "urllib3",
"specs": [
[
"==",
"1.26.19"
]
]
}
],
"lcname": "cloud-governance"
}
JSON
