cloud-governance-test-athiruma


Namecloud-governance-test-athiruma JSON
Version 1.1.50 PyPI version JSON
download
home_pagehttps://github.com/redhat-performance/cloud-governance
SummaryCloud Governance Tool
upload_time2022-12-17 16:16:27
maintainer
docs_urlNone
authorRed Hat
requires_python
licenseApache License 2.0
keywords
VCS
bugtrack_url
requirements attrs azure-identity azure-mgmt-costmanagement azure-mgmt-subscription botocore boto3 elasticsearch elasticsearch-dsl google-api-python-client google-auth-httplib2 google-auth-oauthlib ibm_platform_services myst-parser pandas PyGitHub requests retry SoftLayer sphinx sphinx-rtd-theme python-ldap typing typeguard
Travis-CI No Travis.
coveralls test coverage 
            
[![PyPI Latest Release](https://img.shields.io/pypi/v/cloud-governance.svg)](https://pypi.org/project/cloud-governance/)
[![Container Repository on Quay](https://quay.io/repository/projectquay/quay/status "Container Repository on Quay")](https://quay.io/repository/ebattat/cloud-governance?tab=tags)
[![Actions Status](https://github.com/redhat-performance/cloud-governance/workflows/Build/badge.svg)](https://github.com/redhat-performance/cloud-governance/actions)
[![Coverage Status](https://coveralls.io/repos/github/redhat-performance/cloud-governance/badge.svg?branch=main)](https://coveralls.io/github/redhat-performance/cloud-governance?branch=main)
[![Documentation Status](https://readthedocs.org/projects/cloud-governance/badge/?version=latest)](https://cloud-governance.readthedocs.io/en/latest/?badge=latest)
[![python](https://img.shields.io/pypi/pyversions/cloud-governance.svg?color=%2334D058)](https://pypi.org/project/cloud-governance)
[![License](https://img.shields.io/pypi/l/cloud-governance.svg)](https://github.com/redhat-performance/cloud-governance/blob/main/LICENSE)


# Cloud Governance

![](images/cloud_governance.png)

## What is it?

**Cloud Governance** tool provides a lightweight and flexible framework for deploying cloud management policies focusing on cost optimize and security.

This tool support the following policies:
[policy](cloud_governance/policy)

[AWS Polices](cloud_governance/policy/aws)

* Real time Openshift Cluster cost, User cost
* [ec2_idle](cloud_governance/policy/aws/ec2_idle.py): idle ec2 in last 4 days, cpu < 2% & network < 5mb.
* [ec2_run](cloud_governance/policy/aws/ec2_run.py): running ec2.
* [ebs_unattached](cloud_governance/policy/aws/ebs_unattached.py): volumes that did not connect to instance, volume in available status.
* [ebs_in_use](cloud_governance/policy/aws/ebs_in_use.py): in use volumes.
* [tag_resources](cloud_governance/aws/tag_cluster): Update cluster and non cluster resource tags fetching from the user tags or from the mandatory tags
* [zombie_cluster_resource](cloud_governance/policy/aws/zombie_cluster_resource.py): Delete cluster's zombie resources
* [tag_non_cluster](cloud_governance/aws/tag_non_cluster): tag ec2 resources (instance, volume, ami, snapshot) by instance name
* [tag_iam_user](cloud_governance/aws/tag_user): update the user tags from the csv file
* [cost_explorer](cloud_governance/policy/aws/cost_explorer.py): Get data from cost explorer and upload to ElasticSearch
* [ip_unattached](cloud_governance/policy/aws/ip_unattached.py): Get the unattached IP and delete it after 7 days.
* [s3_inactive](cloud_governance/policy/aws/s3_inactive.py): Get the inactive/empty buckets and delete them after 7 days.
* [empty_roles](cloud_governance/policy/aws/empty_roles.py): Get empty roles and delete it after 7 days.
* [zombie_snapshots](cloud_governance/policy/aws/zombie_snapshots.py): Get the zombie snapshots and delete it after 7 days.
* [nat_gateway_unused](cloud_governance/policy/aws/nat_gateway_unused.py): Get the unused nat gateways and deletes it after 7 days.
* gitleaks: scan Github repository git leak (security scan)  
* [cost_over_usage](cloud_governance/policy/aws/cost_over_usage.py): send mail to aws user if over usage cost

[IBM policies](cloud_governance/policy/ibm)

* [tag_baremetal](cloud_governance/policy/ibm/tag_baremetal.py): Tag IBM baremetal machines
* [tag_vm](cloud_governance/policy/ibm/tag_vm.py): Tga IBM Virtual Machines machines

** You can write your own policy using [Cloud-Custodian](https://cloudcustodian.io/docs/quickstart/index.html)
   and run it (see 'custom cloud custodian policy' in [Policy workflows](#policy-workloads)).


![](images/cloud_governance1.png)
![](images/demo.gif)

![](images/cloud_governance2.png)

Reference:
* The cloud-governance package is placed in [PyPi](https://pypi.org/project/cloud-governance/)
* The cloud-governance container image is placed in [Quay.io](https://quay.io/repository/ebattat/cloud-governance)
* The cloud-governance readthedocs link is [ReadTheDocs](https://cloud-governance.readthedocs.io/en/latest/)
![](images/cloud_governance3.png)

_**Table of Contents**_

<!-- TOC -->
- [Installation](#installation)
- [Configuration](#configuration)
- [Run AWS Policy Using Podman](#run-aws-policy-using-podman)
- [Run IBM Policy Using Podman](#run-ibm-policy-using-podman)
- [Run Policy Using Pod](#run-policy-using-pod)
- [Pytest](#pytest)
- [Post Installation](#post-installation)

<!-- /TOC -->

## Installation

#### Download cloud-governance image from quay.io
```sh
# Need to run it with root privileges
sudo podman pull quay.io/ebattat/cloud-governance
```

#### Environment variables description:

(mandatory)AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID

(mandatory)AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY

##### Policy name:
(mandatory)policy=ec2_idle / ec2_run / ebs_unattached / ebs_in_use / tag_cluster_resource / zombie_cluster_resource / tag_ec2_resource

##### Policy logs output
(mandatory)policy_output=s3://redhat-cloud-governance/logs

##### Cluster or instance name:
(mandatory policy:tag_cluster_resource)resource_name=ocs-test

##### Cluster or instance tags:
(mandatory policy:tag_cluster_resource)mandatory_tags="{'Owner': 'Name','Email': 'name@redhat.com','Purpose': 'test'}"

##### gitleaks
(mandatory policy: gitleaks)git_access_token=$git_access_token
(mandatory policy: gitleaks)git_repo=https://github.com/redhat-performance/cloud-governance
(optional policy: gitleaks)several_repos=yes/no (default = no)

##### Choose a specific region or all for all the regions, default : us-east-2
(optional)AWS_DEFAULT_REGION=us-east-2/all (default = us-east-2)

##### Choose dry run or not, default yes
(optional)dry_run=yes/no (default = yes)

##### Choose log level, default INFO
(optional)log_level=INFO (default = INFO)

#### LDAP hostname to fetch mail records
LDAP_HOST_NAME=ldap.example.com

#### Enable Google Drive API in console and create Service account
GOOGLE_APPLICATION_CREDENTIALS=$pwd/service_account.json

# Configuration

### AWS Configuration

#### Create a user and a bucket
* Create user with IAM [iam](iam/clouds)
* Create a logs bucket [create_bucket.sh](iam/cloud/aws/create_bucket.sh)

### IBM Configuration
* Create classic infrastructure API key

## Run AWS Policy Using Podman 
```sh
# policy=ec2_idle
sudo podman run --rm --name cloud-governance -e policy="ec2_idle" -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" -e AWS_DEFAULT_REGION="us-east-2" -e dry_run="yes" -e policy_output="s3://bucket/logs" -e log_level="INFO" "quay.io/ebattat/cloud-governance"

# policy=ec2_run
sudo podman run --rm --name cloud-governance -e policy="ec2_run" -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" -e AWS_DEFAULT_REGION="us-east-2" -e dry_run="yes" -e policy_output="s3://bucket/logs" -e log_level="INFO" "quay.io/ebattat/cloud-governance"

# select policy ['ec2_stop', 's3_inactive', 'empty_roles', 'ip_unattached', 'nat_gateway_unused', 'zombie_snapshots']
sudo podman run --rm --name cloud-governance -e policy="policy" -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" -e AWS_DEFAULT_REGION="us-east-2" -e dry_run="yes"  -e log_level="INFO" "quay.io/ebattat/cloud-governance"

# policy=ebs_unattached
sudo podman run --rm --name cloud-governance -e policy="ebs_unattached" -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" -e AWS_DEFAULT_REGION="us-east-2" -e dry_run="yes" -e policy_output="s3://bucket/logs" -e log_level="INFO" "quay.io/ebattat/cloud-governance"

# policy=ebs_in_use
sudo podman run --rm --name cloud-governance -e policy="ebs_in_use" -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" -e AWS_DEFAULT_REGION="us-east-2" -e dry_run="yes" -e policy_output="s3://bucket/logs" -e log_level="INFO" "quay.io/ebattat/cloud-governance"

# policy=zombie_cluster_resource
sudo podman run --rm --name cloud-governance -e policy="zombie_cluster_resource" -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" -e AWS_DEFAULT_REGION="us-east-2" -e dry_run="yes" -e resource="zombie_cluster_elastic_ip" -e cluster_tag="kubernetes.io/cluster/test-pd9qq" -e log_level="INFO" "quay.io/ebattat/cloud-governance"

# policy=tag_resources
sudo podman run --rm --name cloud-governance -e policy="tag_resources" -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" -e AWS_DEFAULT_REGION="us-east-2" -e tag_operation="read/update/delete" -e mandatory_tags="{'Owner': 'Name','Email': 'name@redhat.com','Purpose': 'test'}" -e log_level="INFO" -v "/etc/localtime":"/etc/localtime" "quay.io/ebattat/cloud-governance"

# policy=tag_non_cluster
sudo podman run --rm --name cloud-governance -e policy="tag_non_cluster" -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" -e AWS_DEFAULT_REGION="us-east-2" -e tag_operation="read/update/delete" -e mandatory_tags="{'Owner': 'Name','Email': 'name@redhat.com','Purpose': 'test'}" -e log_level="INFO" -v "/etc/localtime":"/etc/localtime" "quay.io/ebattat/cloud-governance"

# policy=tag_iam_user
sudo podman run --rm --name cloud-governance -e policy="tag_iam_user" -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" -e user_tag_operation="read/update/delete" -e remove_tags="['Environment', 'Test']" -e username="test_username" -e file_name="tag_user.csv"  -e log_level="INFO" -v "/home/user/tag_user.csv":"/tmp/tag_user.csv" --privileged "quay.io/ebattat/cloud-governance"

# policy=cost_explorer
sudo podman run --rm --name cloud-governance -e policy="cost_explorer" -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" -e es_host="$elasticsearch_host" -e es_port="$elasticsearch_port" -e es_index="$elasticsearch_index" -e cost_metric=UnblendedCost -e start_date="$start_date" -e end_date="$end_date" -e granularity="DAILY" -e cost_explorer_tags="['User', 'Budget', 'Project', 'Manager', 'Owner', 'LaunchTime', 'Name', 'Email']" -e log_level="INFO" "quay.io/ebattat/cloud-governance:latest"
sudo podman run --rm --name cloud-governance -e policy="cost_explorer" -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" -e es_index="elasticsearch_index" -e cost_metric="UnblendedCost" -e start_date="$start_date" -e end_date="$end_date" -e granularity="DAILY" -e cost_explorer_tags="['User', 'Budget', 'Project', 'Manager', 'Owner', 'LaunchTime', 'Name', 'Email']" -e file_name="cost_explorer.txt" -v "/home/cost_explorer.txt":"/tmp/cost_explorer.txt" -e log_level="INFO" "quay.io/ebattat/cloud-governance:latest"

# policy=validate_iam_user_tags
sudo podman run --rm --name cloud-governance  -e policy="validate_iam_user_tags" -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" -e validate_type="spaces/tags" -e user_tags="['Budget', 'User', 'Owner', 'Manager', 'Environment', 'Project']"   -e log_level="INFO" "quay.io/ebattat/cloud-governance:latest"

# policy=gitleaks
sudo podman run --rm --name cloud-governance -e policy="gitleaks" -e git_access_token="$git_access_token" -e git_repo="https://github.com/redhat-performance/cloud-governance" -e several_repos="no" -e log_level="INFO" "quay.io/ebattat/cloud-governance"

# custom cloud custodian policy (path for custom policy: -v /home/user/custodian_policy:/custodian_policy)
sudo podman run --rm --name cloud-governance -e policy="/custodian_policy/policy.yml" -e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" -e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" -e AWS_DEFAULT_REGION="us-east-2" -e dry_run="yes" -e policy_output="s3://bucket/logs" -e log_level="INFO" -v "/home/user/custodian_policy":"/custodian_policy" --privileged "quay.io/ebattat/cloud-governance"

```

## Run IBM Policy Using Podman

```sh
# policy=tag_baremetal
podman run --rm --name cloud-governance -e policy="tag_baremetal" -e account="$account" -e IBM_API_USERNAME="$IBM_API_USERNAME" -e IBM_API_KEY="$IBM_API_KEY" -e SPREADSHEET_ID="$SPREADSHEET_ID" -e GOOGLE_APPLICATION_CREDENTIALS="$GOOGLE_APPLICATION_CREDENTIALS" -v $GOOGLE_APPLICATION_CREDENTIALS:$GOOGLE_APPLICATION_CREDENTIALS -e LDAP_USER_HOST="$LDAP_USER_HOST" -e tag_operation="update" -e log_level="INFO" -v "/etc/localtime":"/etc/localtime" "quay.io/ebattat/cloud-governance:latest"

# tag=tab_vm
podman run --rm --name cloud-governance -e policy="tag_vm" -e account="$account" -e IBM_API_USERNAME="$IBM_API_USERNAME" -e IBM_API_KEY="$IBM_API_KEY" -e SPREADSHEET_ID="$SPREADSHEET_ID" -e GOOGLE_APPLICATION_CREDENTIALS="$GOOGLE_APPLICATION_CREDENTIALS" -v $GOOGLE_APPLICATION_CREDENTIALS:$GOOGLE_APPLICATION_CREDENTIALS -e LDAP_USER_HOST="$LDAP_USER_HOST" -e tag_operation="update" -e log_level="INFO" -v "/etc/localtime":"/etc/localtime" "quay.io/ebattat/cloud-governance:latest"

```

## Run Policy Using Pod

#### Run as a pod job via OpenShift

Job Pod: [cloud-governance.yaml](pod_yaml/cloud-governance.yaml)

Configmaps: [cloud_governance_configmap.yaml](pod_yaml/cloud_governance_configmap.yaml)

Quay.io Secret: [quayio_secret.sh](pod_yaml/quayio_secret.sh)

AWS Secret: [cloud_governance_secret.yaml](pod_yaml/cloud_governance_secret.yaml)

    * Need to convert secret key to base64 [run_base64.py](pod_yaml/run_base64.py)

## Pytest

##### Cloud-governance integration tests using pytest
```sh
python3 -m venv governance
source governance/bin/activate
(governance) $ python -m pip install --upgrade pip
(governance) $ pip install coverage
(governance) $ pip install pytest
(governance) $ git clone https://github.com/redhat-performance/cloud-governance
(governance) $ cd cloud-governance
(governance) $ coverage run -m pytest
(governance) $ deactivate
rm -rf *governance*
```

## Post Installation

#### Delete cloud-governance image
```sh
sudo podman rmi quay.io/ebattat/cloud-governance
```

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/redhat-performance/cloud-governance",
    "name": "cloud-governance-test-athiruma",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "",
    "author": "Red Hat",
    "author_email": "ebattat@redhat.com, athiruma@redhat.com",
    "download_url": "https://files.pythonhosted.org/packages/8c/4e/f08936c3d34e4fd827f7a34cc85239bed4a7fbdc5c98a2eb820b52f871df/cloud-governance-test-athiruma-1.1.50.tar.gz",
    "platform": null,
    "description": "\n[![PyPI Latest Release](https://img.shields.io/pypi/v/cloud-governance.svg)](https://pypi.org/project/cloud-governance/)\n[![Container Repository on Quay](https://quay.io/repository/projectquay/quay/status \"Container Repository on Quay\")](https://quay.io/repository/ebattat/cloud-governance?tab=tags)\n[![Actions Status](https://github.com/redhat-performance/cloud-governance/workflows/Build/badge.svg)](https://github.com/redhat-performance/cloud-governance/actions)\n[![Coverage Status](https://coveralls.io/repos/github/redhat-performance/cloud-governance/badge.svg?branch=main)](https://coveralls.io/github/redhat-performance/cloud-governance?branch=main)\n[![Documentation Status](https://readthedocs.org/projects/cloud-governance/badge/?version=latest)](https://cloud-governance.readthedocs.io/en/latest/?badge=latest)\n[![python](https://img.shields.io/pypi/pyversions/cloud-governance.svg?color=%2334D058)](https://pypi.org/project/cloud-governance)\n[![License](https://img.shields.io/pypi/l/cloud-governance.svg)](https://github.com/redhat-performance/cloud-governance/blob/main/LICENSE)\n\n\n# Cloud Governance\n\n![](images/cloud_governance.png)\n\n## What is it?\n\n**Cloud Governance** tool provides a lightweight and flexible framework for deploying cloud management policies focusing on cost optimize and security.\n\nThis tool support the following policies:\n[policy](cloud_governance/policy)\n\n[AWS Polices](cloud_governance/policy/aws)\n\n* Real time Openshift Cluster cost, User cost\n* [ec2_idle](cloud_governance/policy/aws/ec2_idle.py): idle ec2 in last 4 days, cpu < 2% & network < 5mb.\n* [ec2_run](cloud_governance/policy/aws/ec2_run.py): running ec2.\n* [ebs_unattached](cloud_governance/policy/aws/ebs_unattached.py): volumes that did not connect to instance, volume in available status.\n* [ebs_in_use](cloud_governance/policy/aws/ebs_in_use.py): in use volumes.\n* [tag_resources](cloud_governance/aws/tag_cluster): Update cluster and non cluster resource tags fetching from the user tags or from the mandatory tags\n* [zombie_cluster_resource](cloud_governance/policy/aws/zombie_cluster_resource.py): Delete cluster's zombie resources\n* [tag_non_cluster](cloud_governance/aws/tag_non_cluster): tag ec2 resources (instance, volume, ami, snapshot) by instance name\n* [tag_iam_user](cloud_governance/aws/tag_user): update the user tags from the csv file\n* [cost_explorer](cloud_governance/policy/aws/cost_explorer.py): Get data from cost explorer and upload to ElasticSearch\n* [ip_unattached](cloud_governance/policy/aws/ip_unattached.py): Get the unattached IP and delete it after 7 days.\n* [s3_inactive](cloud_governance/policy/aws/s3_inactive.py): Get the inactive/empty buckets and delete them after 7 days.\n* [empty_roles](cloud_governance/policy/aws/empty_roles.py): Get empty roles and delete it after 7 days.\n* [zombie_snapshots](cloud_governance/policy/aws/zombie_snapshots.py): Get the zombie snapshots and delete it after 7 days.\n* [nat_gateway_unused](cloud_governance/policy/aws/nat_gateway_unused.py): Get the unused nat gateways and deletes it after 7 days.\n* gitleaks: scan Github repository git leak (security scan)  \n* [cost_over_usage](cloud_governance/policy/aws/cost_over_usage.py): send mail to aws user if over usage cost\n\n[IBM policies](cloud_governance/policy/ibm)\n\n* [tag_baremetal](cloud_governance/policy/ibm/tag_baremetal.py): Tag IBM baremetal machines\n* [tag_vm](cloud_governance/policy/ibm/tag_vm.py): Tga IBM Virtual Machines machines\n\n** You can write your own policy using [Cloud-Custodian](https://cloudcustodian.io/docs/quickstart/index.html)\n   and run it (see 'custom cloud custodian policy' in [Policy workflows](#policy-workloads)).\n\n\n![](images/cloud_governance1.png)\n![](images/demo.gif)\n\n![](images/cloud_governance2.png)\n\nReference:\n* The cloud-governance package is placed in [PyPi](https://pypi.org/project/cloud-governance/)\n* The cloud-governance container image is placed in [Quay.io](https://quay.io/repository/ebattat/cloud-governance)\n* The cloud-governance readthedocs link is [ReadTheDocs](https://cloud-governance.readthedocs.io/en/latest/)\n![](images/cloud_governance3.png)\n\n_**Table of Contents**_\n\n<!-- TOC -->\n- [Installation](#installation)\n- [Configuration](#configuration)\n- [Run AWS Policy Using Podman](#run-aws-policy-using-podman)\n- [Run IBM Policy Using Podman](#run-ibm-policy-using-podman)\n- [Run Policy Using Pod](#run-policy-using-pod)\n- [Pytest](#pytest)\n- [Post Installation](#post-installation)\n\n<!-- /TOC -->\n\n## Installation\n\n#### Download cloud-governance image from quay.io\n```sh\n# Need to run it with root privileges\nsudo podman pull quay.io/ebattat/cloud-governance\n```\n\n#### Environment variables description:\n\n(mandatory)AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID\n\n(mandatory)AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY\n\n##### Policy name:\n(mandatory)policy=ec2_idle / ec2_run / ebs_unattached / ebs_in_use / tag_cluster_resource / zombie_cluster_resource / tag_ec2_resource\n\n##### Policy logs output\n(mandatory)policy_output=s3://redhat-cloud-governance/logs\n\n##### Cluster or instance name:\n(mandatory policy:tag_cluster_resource)resource_name=ocs-test\n\n##### Cluster or instance tags:\n(mandatory policy:tag_cluster_resource)mandatory_tags=\"{'Owner': 'Name','Email': 'name@redhat.com','Purpose': 'test'}\"\n\n##### gitleaks\n(mandatory policy: gitleaks)git_access_token=$git_access_token\n(mandatory policy: gitleaks)git_repo=https://github.com/redhat-performance/cloud-governance\n(optional policy: gitleaks)several_repos=yes/no (default = no)\n\n##### Choose a specific region or all for all the regions, default : us-east-2\n(optional)AWS_DEFAULT_REGION=us-east-2/all (default = us-east-2)\n\n##### Choose dry run or not, default yes\n(optional)dry_run=yes/no (default = yes)\n\n##### Choose log level, default INFO\n(optional)log_level=INFO (default = INFO)\n\n#### LDAP hostname to fetch mail records\nLDAP_HOST_NAME=ldap.example.com\n\n#### Enable Google Drive API in console and create Service account\nGOOGLE_APPLICATION_CREDENTIALS=$pwd/service_account.json\n\n# Configuration\n\n### AWS Configuration\n\n#### Create a user and a bucket\n* Create user with IAM [iam](iam/clouds)\n* Create a logs bucket [create_bucket.sh](iam/cloud/aws/create_bucket.sh)\n\n### IBM Configuration\n* Create classic infrastructure API key\n\n## Run AWS Policy Using Podman \n```sh\n# policy=ec2_idle\nsudo podman run --rm --name cloud-governance -e policy=\"ec2_idle\" -e AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\" -e AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\" -e AWS_DEFAULT_REGION=\"us-east-2\" -e dry_run=\"yes\" -e policy_output=\"s3://bucket/logs\" -e log_level=\"INFO\" \"quay.io/ebattat/cloud-governance\"\n\n# policy=ec2_run\nsudo podman run --rm --name cloud-governance -e policy=\"ec2_run\" -e AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\" -e AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\" -e AWS_DEFAULT_REGION=\"us-east-2\" -e dry_run=\"yes\" -e policy_output=\"s3://bucket/logs\" -e log_level=\"INFO\" \"quay.io/ebattat/cloud-governance\"\n\n# select policy ['ec2_stop', 's3_inactive', 'empty_roles', 'ip_unattached', 'nat_gateway_unused', 'zombie_snapshots']\nsudo podman run --rm --name cloud-governance -e policy=\"policy\" -e AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\" -e AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\" -e AWS_DEFAULT_REGION=\"us-east-2\" -e dry_run=\"yes\"  -e log_level=\"INFO\" \"quay.io/ebattat/cloud-governance\"\n\n# policy=ebs_unattached\nsudo podman run --rm --name cloud-governance -e policy=\"ebs_unattached\" -e AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\" -e AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\" -e AWS_DEFAULT_REGION=\"us-east-2\" -e dry_run=\"yes\" -e policy_output=\"s3://bucket/logs\" -e log_level=\"INFO\" \"quay.io/ebattat/cloud-governance\"\n\n# policy=ebs_in_use\nsudo podman run --rm --name cloud-governance -e policy=\"ebs_in_use\" -e AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\" -e AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\" -e AWS_DEFAULT_REGION=\"us-east-2\" -e dry_run=\"yes\" -e policy_output=\"s3://bucket/logs\" -e log_level=\"INFO\" \"quay.io/ebattat/cloud-governance\"\n\n# policy=zombie_cluster_resource\nsudo podman run --rm --name cloud-governance -e policy=\"zombie_cluster_resource\" -e AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\" -e AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\" -e AWS_DEFAULT_REGION=\"us-east-2\" -e dry_run=\"yes\" -e resource=\"zombie_cluster_elastic_ip\" -e cluster_tag=\"kubernetes.io/cluster/test-pd9qq\" -e log_level=\"INFO\" \"quay.io/ebattat/cloud-governance\"\n\n# policy=tag_resources\nsudo podman run --rm --name cloud-governance -e policy=\"tag_resources\" -e AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\" -e AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\" -e AWS_DEFAULT_REGION=\"us-east-2\" -e tag_operation=\"read/update/delete\" -e mandatory_tags=\"{'Owner': 'Name','Email': 'name@redhat.com','Purpose': 'test'}\" -e log_level=\"INFO\" -v \"/etc/localtime\":\"/etc/localtime\" \"quay.io/ebattat/cloud-governance\"\n\n# policy=tag_non_cluster\nsudo podman run --rm --name cloud-governance -e policy=\"tag_non_cluster\" -e AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\" -e AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\" -e AWS_DEFAULT_REGION=\"us-east-2\" -e tag_operation=\"read/update/delete\" -e mandatory_tags=\"{'Owner': 'Name','Email': 'name@redhat.com','Purpose': 'test'}\" -e log_level=\"INFO\" -v \"/etc/localtime\":\"/etc/localtime\" \"quay.io/ebattat/cloud-governance\"\n\n# policy=tag_iam_user\nsudo podman run --rm --name cloud-governance -e policy=\"tag_iam_user\" -e AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\" -e AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\" -e user_tag_operation=\"read/update/delete\" -e remove_tags=\"['Environment', 'Test']\" -e username=\"test_username\" -e file_name=\"tag_user.csv\"  -e log_level=\"INFO\" -v \"/home/user/tag_user.csv\":\"/tmp/tag_user.csv\" --privileged \"quay.io/ebattat/cloud-governance\"\n\n# policy=cost_explorer\nsudo podman run --rm --name cloud-governance -e policy=\"cost_explorer\" -e AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\" -e AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\" -e es_host=\"$elasticsearch_host\" -e es_port=\"$elasticsearch_port\" -e es_index=\"$elasticsearch_index\" -e cost_metric=UnblendedCost -e start_date=\"$start_date\" -e end_date=\"$end_date\" -e granularity=\"DAILY\" -e cost_explorer_tags=\"['User', 'Budget', 'Project', 'Manager', 'Owner', 'LaunchTime', 'Name', 'Email']\" -e log_level=\"INFO\" \"quay.io/ebattat/cloud-governance:latest\"\nsudo podman run --rm --name cloud-governance -e policy=\"cost_explorer\" -e AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\" -e AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\" -e es_index=\"elasticsearch_index\" -e cost_metric=\"UnblendedCost\" -e start_date=\"$start_date\" -e end_date=\"$end_date\" -e granularity=\"DAILY\" -e cost_explorer_tags=\"['User', 'Budget', 'Project', 'Manager', 'Owner', 'LaunchTime', 'Name', 'Email']\" -e file_name=\"cost_explorer.txt\" -v \"/home/cost_explorer.txt\":\"/tmp/cost_explorer.txt\" -e log_level=\"INFO\" \"quay.io/ebattat/cloud-governance:latest\"\n\n# policy=validate_iam_user_tags\nsudo podman run --rm --name cloud-governance  -e policy=\"validate_iam_user_tags\" -e AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\" -e AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\" -e validate_type=\"spaces/tags\" -e user_tags=\"['Budget', 'User', 'Owner', 'Manager', 'Environment', 'Project']\"   -e log_level=\"INFO\" \"quay.io/ebattat/cloud-governance:latest\"\n\n# policy=gitleaks\nsudo podman run --rm --name cloud-governance -e policy=\"gitleaks\" -e git_access_token=\"$git_access_token\" -e git_repo=\"https://github.com/redhat-performance/cloud-governance\" -e several_repos=\"no\" -e log_level=\"INFO\" \"quay.io/ebattat/cloud-governance\"\n\n# custom cloud custodian policy (path for custom policy: -v /home/user/custodian_policy:/custodian_policy)\nsudo podman run --rm --name cloud-governance -e policy=\"/custodian_policy/policy.yml\" -e AWS_ACCESS_KEY_ID=\"$AWS_ACCESS_KEY_ID\" -e AWS_SECRET_ACCESS_KEY=\"$AWS_SECRET_ACCESS_KEY\" -e AWS_DEFAULT_REGION=\"us-east-2\" -e dry_run=\"yes\" -e policy_output=\"s3://bucket/logs\" -e log_level=\"INFO\" -v \"/home/user/custodian_policy\":\"/custodian_policy\" --privileged \"quay.io/ebattat/cloud-governance\"\n\n```\n\n## Run IBM Policy Using Podman\n\n```sh\n# policy=tag_baremetal\npodman run --rm --name cloud-governance -e policy=\"tag_baremetal\" -e account=\"$account\" -e IBM_API_USERNAME=\"$IBM_API_USERNAME\" -e IBM_API_KEY=\"$IBM_API_KEY\" -e SPREADSHEET_ID=\"$SPREADSHEET_ID\" -e GOOGLE_APPLICATION_CREDENTIALS=\"$GOOGLE_APPLICATION_CREDENTIALS\" -v $GOOGLE_APPLICATION_CREDENTIALS:$GOOGLE_APPLICATION_CREDENTIALS -e LDAP_USER_HOST=\"$LDAP_USER_HOST\" -e tag_operation=\"update\" -e log_level=\"INFO\" -v \"/etc/localtime\":\"/etc/localtime\" \"quay.io/ebattat/cloud-governance:latest\"\n\n# tag=tab_vm\npodman run --rm --name cloud-governance -e policy=\"tag_vm\" -e account=\"$account\" -e IBM_API_USERNAME=\"$IBM_API_USERNAME\" -e IBM_API_KEY=\"$IBM_API_KEY\" -e SPREADSHEET_ID=\"$SPREADSHEET_ID\" -e GOOGLE_APPLICATION_CREDENTIALS=\"$GOOGLE_APPLICATION_CREDENTIALS\" -v $GOOGLE_APPLICATION_CREDENTIALS:$GOOGLE_APPLICATION_CREDENTIALS -e LDAP_USER_HOST=\"$LDAP_USER_HOST\" -e tag_operation=\"update\" -e log_level=\"INFO\" -v \"/etc/localtime\":\"/etc/localtime\" \"quay.io/ebattat/cloud-governance:latest\"\n\n```\n\n## Run Policy Using Pod\n\n#### Run as a pod job via OpenShift\n\nJob Pod: [cloud-governance.yaml](pod_yaml/cloud-governance.yaml)\n\nConfigmaps: [cloud_governance_configmap.yaml](pod_yaml/cloud_governance_configmap.yaml)\n\nQuay.io Secret: [quayio_secret.sh](pod_yaml/quayio_secret.sh)\n\nAWS Secret: [cloud_governance_secret.yaml](pod_yaml/cloud_governance_secret.yaml)\n\n    * Need to convert secret key to base64 [run_base64.py](pod_yaml/run_base64.py)\n\n## Pytest\n\n##### Cloud-governance integration tests using pytest\n```sh\npython3 -m venv governance\nsource governance/bin/activate\n(governance) $ python -m pip install --upgrade pip\n(governance) $ pip install coverage\n(governance) $ pip install pytest\n(governance) $ git clone https://github.com/redhat-performance/cloud-governance\n(governance) $ cd cloud-governance\n(governance) $ coverage run -m pytest\n(governance) $ deactivate\nrm -rf *governance*\n```\n\n## Post Installation\n\n#### Delete cloud-governance image\n```sh\nsudo podman rmi quay.io/ebattat/cloud-governance\n```\n\n\n",
    "bugtrack_url": null,
    "license": "Apache License 2.0",
    "summary": "Cloud Governance Tool",
    "version": "1.1.50",
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "4064eb43a4c364a78d2ca6bcca23e784",
                "sha256": "88408222cffbf57fdffbbb4cda900399f8db93970b041e212c297a5dc5ca5903"
            },
            "downloads": -1,
            "filename": "cloud-governance-test-athiruma-1.1.50.tar.gz",
            "has_sig": false,
            "md5_digest": "4064eb43a4c364a78d2ca6bcca23e784",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 107441,
            "upload_time": "2022-12-17T16:16:27",
            "upload_time_iso_8601": "2022-12-17T16:16:27.478859Z",
            "url": "https://files.pythonhosted.org/packages/8c/4e/f08936c3d34e4fd827f7a34cc85239bed4a7fbdc5c98a2eb820b52f871df/cloud-governance-test-athiruma-1.1.50.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2022-12-17 16:16:27",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "redhat-performance",
    "github_project": "cloud-governance",
    "travis_ci": false,
    "coveralls": true,
    "github_actions": true,
    "requirements": [
        {
            "name": "attrs",
            "specs": [
                [
                    "==",
                    "21.4.0"
                ]
            ]
        },
        {
            "name": "azure-identity",
            "specs": [
                [
                    "==",
                    "1.12.0"
                ]
            ]
        },
        {
            "name": "azure-mgmt-costmanagement",
            "specs": [
                [
                    "==",
                    "3.0.0"
                ]
            ]
        },
        {
            "name": "azure-mgmt-subscription",
            "specs": [
                [
                    "==",
                    "3.1.1"
                ]
            ]
        },
        {
            "name": "botocore",
            "specs": [
                [
                    "==",
                    "1.29.1"
                ]
            ]
        },
        {
            "name": "boto3",
            "specs": [
                [
                    "==",
                    "1.26.1"
                ]
            ]
        },
        {
            "name": "elasticsearch",
            "specs": [
                [
                    "==",
                    "7.11.0"
                ]
            ]
        },
        {
            "name": "elasticsearch-dsl",
            "specs": [
                [
                    "==",
                    "7.4.0"
                ]
            ]
        },
        {
            "name": "google-api-python-client",
            "specs": [
                [
                    "==",
                    "2.57.0"
                ]
            ]
        },
        {
            "name": "google-auth-httplib2",
            "specs": [
                [
                    "==",
                    "0.1.0"
                ]
            ]
        },
        {
            "name": "google-auth-oauthlib",
            "specs": [
                [
                    "==",
                    "0.5.2"
                ]
            ]
        },
        {
            "name": "ibm_platform_services",
            "specs": [
                [
                    "==",
                    "0.27.0"
                ]
            ]
        },
        {
            "name": "myst-parser",
            "specs": [
                [
                    "==",
                    "0.17.0"
                ]
            ]
        },
        {
            "name": "pandas",
            "specs": []
        },
        {
            "name": "PyGitHub",
            "specs": [
                [
                    "==",
                    "1.55"
                ]
            ]
        },
        {
            "name": "requests",
            "specs": [
                [
                    "==",
                    "2.27.1"
                ]
            ]
        },
        {
            "name": "retry",
            "specs": [
                [
                    "==",
                    "0.9.2"
                ]
            ]
        },
        {
            "name": "SoftLayer",
            "specs": [
                [
                    "==",
                    "6.0.0"
                ]
            ]
        },
        {
            "name": "sphinx",
            "specs": [
                [
                    "==",
                    "4.5.0"
                ]
            ]
        },
        {
            "name": "sphinx-rtd-theme",
            "specs": [
                [
                    "==",
                    "1.0.0"
                ]
            ]
        },
        {
            "name": "python-ldap",
            "specs": [
                [
                    "==",
                    "3.4.2"
                ]
            ]
        },
        {
            "name": "typing",
            "specs": [
                [
                    "==",
                    "3.7.4.3"
                ]
            ]
        },
        {
            "name": "typeguard",
            "specs": [
                [
                    "==",
                    "2.13.3"
                ]
            ]
        }
    ],
    "lcname": "cloud-governance-test-athiruma"
}
Red Hat
Elapsed time: 0.02117s